https://hubdatai.com/ A personal, non-commercial blog where I share what I have learned in IT support: runbooks, troubleshooting notes, cheat sheets and videos for Azure, AWS, OCI, data platforms, databases and GenAI. Free to read, free to use. https://hubdatai.com/posts/databricks-jobs-for-each-if-else-tasks-job-parameters/ https://hubdatai.com/posts/databricks-jobs-for-each-if-else-tasks-job-parameters/ Thu, 29 Oct 2026 00:00:00 GMT Databricks on Azure Build control flow inside Databricks Jobs instead of an external orchestrator. Pass values between tasks, branch with if/else condition tasks, fan out with for each tasks and concurrency limits, use run-if rules for cleanup, and define it all in an asset bundle. https://hubdatai.com/posts/rag-query-rewriting-multi-query-hyde-decomposition/ https://hubdatai.com/posts/rag-query-rewriting-multi-query-hyde-decomposition/ Thu, 29 Oct 2026 00:00:00 GMT AI & GenAI Improve recall on vague, conversational and multi-part questions by transforming the query before retrieval. Implement standalone-question rewriting, multi-query with reciprocal rank fusion, HyDE and decomposition, then prove the gain with a retrieval evaluation set and keep latency in budget. https://hubdatai.com/posts/airflow-dynamic-task-mapping-assets-data-aware-scheduling/ https://hubdatai.com/posts/airflow-dynamic-task-mapping-assets-data-aware-scheduling/ Thu, 29 Oct 2026 00:00:00 GMT Data Platform Replace copy-pasted tasks and sensor chains with dynamic task mapping and dataset (asset) triggered DAGs. Control fan-out with max_active_tis_per_dag and pools, handle partial failures, and avoid the mapping patterns that overload the scheduler. https://hubdatai.com/posts/airflow-basics-first-dag-operators-schedules/ https://hubdatai.com/posts/airflow-basics-first-dag-operators-schedules/ Thu, 29 Oct 2026 00:00:00 GMT Data Platform Write a small, correct Airflow DAG with the TaskFlow API, set a schedule without the classic start_date and catchup surprises, pass data between tasks, add retries, and test it with airflow dags test before deploying. https://hubdatai.com/posts/aks-azure-cni-overlay-vs-pod-subnet-ip-planning/ https://hubdatai.com/posts/aks-azure-cni-overlay-vs-pod-subnet-ip-planning/ Thu, 29 Oct 2026 00:00:00 GMT Azure How Azure CNI Overlay, flat Azure CNI with a dedicated pod subnet and kubenet differ in IP use, routing and limits; how to size CIDRs for growth; and how to plan egress so clusters do not hit SNAT or IP exhaustion later. https://hubdatai.com/posts/aks-scheduling-topology-spread-taints-affinity-priority/ https://hubdatai.com/posts/aks-scheduling-topology-spread-taints-affinity-priority/ Thu, 29 Oct 2026 00:00:00 GMT Azure Control where pods land on AKS so a zone or node loss does not take a service down. Topology spread constraints, dedicated node pools with taints, pod anti-affinity, priority and preemption, and how they interact with the cluster autoscaler. https://hubdatai.com/posts/aws-s3-basics-buckets-objects-sync-presigned-urls/ https://hubdatai.com/posts/aws-s3-basics-buckets-objects-sync-presigned-urls/ Thu, 29 Oct 2026 00:00:00 GMT AWS Create a private bucket with sensible defaults, upload, download and sync folders, share a single file safely with a presigned URL, and turn on versioning so mistakes can be undone. https://hubdatai.com/posts/iceberg-partition-evolution-hidden-partitioning/ https://hubdatai.com/posts/iceberg-partition-evolution-hidden-partitioning/ Thu, 29 Oct 2026 00:00:00 GMT Data Platform How Iceberg partition transforms work, why queries do not need partition columns, how to evolve a table from daily to hourly or add bucketing without rewriting history, and how to check pruning and compact old layouts when you want to. https://hubdatai.com/posts/oci-events-functions-automated-remediation/ https://hubdatai.com/posts/oci-events-functions-automated-remediation/ Thu, 29 Oct 2026 00:00:00 GMT OCI Build an event-driven remediation loop. An Events rule catches a risky change, an OCI Function using resource principals fixes it and posts a notification. Includes IAM policies, a Python function, testing and the safety rails that stop automation fighting your engineers. https://hubdatai.com/posts/databricks-terraform-provider-workspace-unity-catalog-automation/ https://hubdatai.com/posts/databricks-terraform-provider-workspace-unity-catalog-automation/ Thu, 29 Oct 2026 00:00:00 GMT Databricks on Azure Split Azure Databricks Terraform into account, workspace and data layers, configure providers with Entra ID auth, create storage credentials, external locations, catalogs and grants, and avoid the state and permission traps that make plans destructive. https://hubdatai.com/posts/oci-autonomous-database-refreshable-clones-cloning-strategy/ https://hubdatai.com/posts/oci-autonomous-database-refreshable-clones-cloning-strategy/ Thu, 29 Oct 2026 00:00:00 GMT OCI Choose between full clones, metadata clones, clones from backup and refreshable clones on Autonomous Database. Create and refresh them from the CLI, mask data for test, use refreshable clones for reporting offload, and know the limits that bite. https://hubdatai.com/posts/kubernetes-llm-autoscaling-keda-queue-depth-gpu/ https://hubdatai.com/posts/kubernetes-llm-autoscaling-keda-queue-depth-gpu/ Thu, 29 Oct 2026 00:00:00 GMT AI & GenAI CPU-based HPA does not work for GPU inference. Scale vLLM or TGI replicas on waiting requests and KV cache usage with KEDA and Prometheus, pair it with the cluster autoscaler or Karpenter for GPU nodes, and hide multi-minute cold starts with warm capacity. https://hubdatai.com/posts/aws-cli-v2-first-steps-profiles-sso-login/ https://hubdatai.com/posts/aws-cli-v2-first-steps-profiles-sso-login/ Thu, 29 Oct 2026 00:00:00 GMT AWS Install AWS CLI v2, sign in with IAM Identity Center instead of long-lived keys, use named profiles for several accounts, and always confirm the account and region before you run a change. https://hubdatai.com/posts/aws-iam-abac-tag-based-access-control-conditions/ https://hubdatai.com/posts/aws-iam-abac-tag-based-access-control-conditions/ Thu, 29 Oct 2026 00:00:00 GMT AWS Replace dozens of per-project policies with a few attribute-based policies that compare principal tags with resource tags, protect the tags themselves, pass tags from IAM Identity Center, and test the result before rollout. https://hubdatai.com/posts/aws-iam-basics-users-roles-policies-explained/ https://hubdatai.com/posts/aws-iam-basics-users-roles-policies-explained/ Thu, 29 Oct 2026 00:00:00 GMT AWS The IAM building blocks in plain terms, how a policy document is read, how roles are assumed, and a small worked example that gives an EC2 instance read access to one S3 bucket without keys. https://hubdatai.com/posts/aws-privatelink-endpoint-service-cross-account-nlb/ https://hubdatai.com/posts/aws-privatelink-endpoint-service-cross-account-nlb/ Thu, 29 Oct 2026 00:00:00 GMT AWS Publish an internal service to other VPCs and accounts with PrivateLink instead of peering. Build the NLB-backed endpoint service, approve consumers, set up private DNS with domain verification, and debug the usual connection failures. https://hubdatai.com/posts/azcopy-basics-upload-download-sync-blob-storage/ https://hubdatai.com/posts/azcopy-basics-upload-download-sync-blob-storage/ Thu, 29 Oct 2026 00:00:00 GMT Azure Install AzCopy, sign in with Entra ID instead of keys, copy and sync folders to and from Blob Storage, and read the job logs when a transfer fails part-way. https://hubdatai.com/posts/azure-cli-first-steps-login-subscriptions-defaults-output/ https://hubdatai.com/posts/azure-cli-first-steps-login-subscriptions-defaults-output/ Thu, 29 Oct 2026 00:00:00 GMT Azure A first-day guide to the Azure CLI. Sign in safely, make sure you are in the right subscription, set a default resource group and location, and filter output with --query so you read only what matters. https://hubdatai.com/posts/azure-cost-management-budgets-alerts-basics/ https://hubdatai.com/posts/azure-cost-management-budgets-alerts-basics/ Thu, 29 Oct 2026 00:00:00 GMT Azure Find out what costs money in a subscription, group spend by resource group and tag, create a budget with email alerts, and export daily cost data so surprises show up in days, not at month end. https://hubdatai.com/posts/azure-databricks-cost-tags-missing-azure-cost-management/ https://hubdatai.com/posts/azure-databricks-cost-tags-missing-azure-cost-management/ Thu, 29 Oct 2026 00:00:00 GMT Databricks on Azure Databricks spend shows as untagged in Azure cost analysis or chargeback reports. Find which tag path failed (pool, workspace, conflict, restart, serverless) and fix it. https://hubdatai.com/posts/azure-databricks-first-steps-workspace-compute-notebooks/ https://hubdatai.com/posts/azure-databricks-first-steps-workspace-compute-notebooks/ Thu, 29 Oct 2026 00:00:00 GMT Databricks on Azure A guided first hour in an Azure Databricks workspace. What each sidebar area is for, which compute to pick (serverless, all-purpose, job, SQL warehouse), creating a small cluster with auto-termination, and a first notebook that reads and writes a table. https://hubdatai.com/posts/azure-monitor-dcr-transformations-filter-ingestion-kql/ https://hubdatai.com/posts/azure-monitor-dcr-transformations-filter-ingestion-kql/ Thu, 29 Oct 2026 00:00:00 GMT Azure Use data collection rule transformations to drop noisy rows, remove or hash sensitive columns, and route data to cheaper tables before it is stored, with a safe test-and-rollout process. https://hubdatai.com/posts/azure-virtual-network-manager-security-admin-rules/ https://hubdatai.com/posts/azure-virtual-network-manager-security-admin-rules/ Thu, 29 Oct 2026 00:00:00 GMT Azure Use Azure Virtual Network Manager network groups and security admin rules to enforce organisation-wide allow/deny rules across many VNets, understand how they combine with NSGs, and deploy them without cutting off production. https://hubdatai.com/posts/azure-vm-stop-vs-deallocate-start-resize-basics/ https://hubdatai.com/posts/azure-vm-stop-vs-deallocate-start-resize-basics/ Thu, 29 Oct 2026 00:00:00 GMT Azure What each VM power state means, why "Stopped" can still bill for compute, how to deallocate, start and resize from the CLI and PowerShell, and what changes (IPs, temp disk) when you do. https://hubdatai.com/posts/unity-catalog-lineage-impact-analysis-before-changing-tables/ https://hubdatai.com/posts/unity-catalog-lineage-impact-analysis-before-changing-tables/ Thu, 29 Oct 2026 00:00:00 GMT Databricks on Azure Use system.access.table_lineage and column_lineage to find every job, pipeline, dashboard and user that reads a table or column before you change it, and what lineage misses. https://hubdatai.com/posts/bicep-what-if-deployment-stacks-deny-settings/ https://hubdatai.com/posts/bicep-what-if-deployment-stacks-deny-settings/ Thu, 29 Oct 2026 00:00:00 GMT Azure Use what-if output as a pipeline gate, manage resource lifecycles with deployment stacks so removed resources are actually deleted or detached, and protect stack-managed resources from portal edits with deny settings. https://hubdatai.com/posts/gpu-driver-upgrade-kubernetes-inference-node-pools-checklist/ https://hubdatai.com/posts/gpu-driver-upgrade-kubernetes-inference-node-pools-checklist/ Thu, 29 Oct 2026 00:00:00 GMT AI & GenAI A step-by-step checklist for rolling a new NVIDIA driver across GPU node pools serving LLMs - compatibility checks, capacity headroom, GPU Operator upgrade policy, canary nodes, validation and rollback. https://hubdatai.com/posts/aws-cloudformation-stacksets-organization-service-managed/ https://hubdatai.com/posts/aws-cloudformation-stacksets-organization-service-managed/ Thu, 29 Oct 2026 00:00:00 GMT AWS Deploy baseline resources (roles, Config rules, alarms, logging) to every account and region with service-managed StackSets, roll out with concurrency and failure tolerance, handle drift and failed instances, and avoid the classic StackSet traps. https://hubdatai.com/posts/aws-cloudwatch-first-alarm-sns-email-basics/ https://hubdatai.com/posts/aws-cloudwatch-first-alarm-sns-email-basics/ Thu, 29 Oct 2026 00:00:00 GMT AWS Understand metrics, statistics and periods, create a CPU and a status-check alarm that emails you through SNS, test it on purpose, and avoid the beginner settings that make alarms noisy or silent. https://hubdatai.com/posts/file-formats-basics-csv-json-parquet-avro/ https://hubdatai.com/posts/file-formats-basics-csv-json-parquet-avro/ Thu, 29 Oct 2026 00:00:00 GMT Data Platform What each common data file format is good and bad at, why Parquet is the default for analytics, where Avro and JSON still fit, and how to inspect and convert files with Python, Spark and DuckDB.