Python Scripts in Power Query: Data Transformation Powerhouse<\/strong><\/h2>\n\n\n\nThis is where most people start, and for good reason. Power Query’s Python integration lets you leverage pandas, numpy, and other data manipulation libraries directly in your ETL process.<\/p>\n\n\n\n
Here’s a practical example of cleaning and feature engineering customer data:<\/p>\n\n\n\n
# Power Query automatically provides a ‘dataset’ as a pandas DataFrame<\/em><\/p>\n\n\n\nimport pandas as pd<\/em><\/p>\n\n\n\nimport numpy as np<\/em><\/p>\n\n\n\nfrom sklearn.preprocessing import StandardScaler<\/em><\/p>\n\n\n\n# Clean missing values with business logic<\/em><\/p>\n\n\n\ndataset[‘customer_age’] = dataset[‘customer_age’].fillna(dataset[‘customer_age’].median())<\/em><\/p>\n\n\n\ndataset[‘income’] = dataset[‘income’].fillna(dataset.groupby(‘segment’)[‘income’].transform(‘median’))<\/em><\/p>\n\n\n\n# Feature engineering<\/em><\/p>\n\n\n\ndataset[‘clv_score’] = (dataset[‘total_purchases’] * dataset[‘avg_order_value’]) \/ dataset[‘customer_age’]<\/em><\/p>\n\n\n\ndataset[‘days_since_last_purchase’] = (pd.Timestamp.now() – pd.to_datetime(dataset[‘last_purchase_date’])).dt.days<\/em><\/p>\n\n\n\n# Create customer segments using clustering<\/em><\/p>\n\n\n\nfrom sklearn.cluster import KMeans<\/em><\/p>\n\n\n\nfeatures = [‘income’, ‘total_purchases’, ‘days_since_last_purchase’]<\/em><\/p>\n\n\n\nscaler = StandardScaler()<\/em><\/p>\n\n\n\nscaled_features = scaler.fit_transform(dataset[features].fillna(0))<\/em><\/p>\n\n\n\nkmeans = KMeans(n_clusters=4, random_state=42)<\/em><\/p>\n\n\n\ndataset[‘customer_segment’] = kmeans.fit_predict(scaled_features)<\/em><\/p>\n\n\n\n# Add descriptive segment names<\/em><\/p>\n\n\n\nsegment_map = {0: ‘High Value’, 1: ‘At Risk’, 2: ‘New Customer’, 3: ‘Low Value’}<\/em><\/p>\n\n\n\ndataset[‘segment_name’] = dataset[‘customer_segment’].map(segment_map)<\/em><\/p>\n\n\n\nThe beauty here is that this runs once during data refresh and gets cached. Your end users see the enriched data without knowing complex ML happened behind the scenes.<\/p>\n\n\n\n
Power Query Python Best Practices:<\/strong><\/p>\n\n\n\n\n- Always return the modified dataset DataFrame<\/li>\n\n\n\n
- Handle missing values explicitly<\/li>\n\n\n\n
- Use vectorized operations (pandas\/numpy) for performance<\/li>\n\n\n\n
- Avoid loops when possible<\/li>\n\n\n\n
- Test scripts outside Power BI first<\/li>\n<\/ul>\n\n\n\n
R in Power Query: Statistical Data Prep<\/strong><\/h2>\n\n\n\nR shines in Power Query for statistical transformations and time series preparation:<\/p>\n\n\n\n
# Power Query provides ‘dataset’ as a data frame<\/em><\/p>\n\n\n\nlibrary(dplyr)<\/em><\/p>\n\n\n\nlibrary(forecast)<\/em><\/p>\n\n\n\nlibrary(zoo)<\/em><\/p>\n\n\n\n# Time series cleaning and decomposition<\/em><\/p>\n\n\n\ndataset$date <- as.Date(dataset$date)<\/em><\/p>\n\n\n\ndataset <- dataset %>% arrange(date)<\/em><\/p>\n\n\n\n# Handle missing values in time series<\/em><\/p>\n\n\n\ndataset$sales <- na.fill(dataset$sales, “extend”)<\/em><\/p>\n\n\n\n# Create rolling averages and trends<\/em><\/p>\n\n\n\ndataset$sales_ma_7 <- rollmean(dataset$sales, k=7, fill=NA, align=”right”)<\/em><\/p>\n\n\n\ndataset$sales_ma_30 <- rollmean(dataset$sales, k=30, fill=NA, align=”right”)<\/em><\/p>\n\n\n\n# Seasonal decomposition for forecasting prep<\/em><\/p>\n\n\n\nts_data <- ts(dataset$sales, frequency=12) # Monthly data<\/em><\/p>\n\n\n\ndecomp <- decompose(ts_data, type=”multiplicative”)<\/em><\/p>\n\n\n\ndataset$trend <- as.numeric(decomp$trend)<\/em><\/p>\n\n\n\ndataset$seasonal <- as.numeric(decomp$seasonal)<\/em><\/p>\n\n\n\ndataset$residual <- as.numeric(decomp$random)<\/em><\/p>\n\n\n\n# Calculate growth rates<\/em><\/p>\n\n\n\ndataset$mom_growth <- (dataset$sales \/ lag(dataset$sales) – 1) * 100<\/em><\/p>\n\n\n\ndataset$yoy_growth <- (dataset$sales \/ lag(dataset$sales, 12) – 1) * 100<\/em><\/p>\n\n\n\nPython Visuals: Beyond Standard Charts<\/strong><\/h2>\n\n\n\nPython visuals let you create sophisticated charts that Power BI’s built-in visuals can’t handle. Think heatmaps, statistical plots, or custom business-specific visualizations.<\/p>\n\n\n\n
Here’s a correlation heatmap with clustering:<\/p>\n\n\n\n
import matplotlib.pyplot as plt<\/em><\/p>\n\n\n\nimport seaborn as sns<\/em><\/p>\n\n\n\nimport pandas as pd<\/em><\/p>\n\n\n\nfrom scipy.cluster.hierarchy import dendrogram, linkage<\/em><\/p>\n\n\n\nfrom scipy.spatial.distance import squareform<\/em><\/p>\n\n\n\n# dataset is automatically available<\/em><\/p>\n\n\n\nnumeric_cols = dataset.select_dtypes(include=[np.number]).columns<\/em><\/p>\n\n\n\ncorr_matrix = dataset[numeric_cols].corr()<\/em><\/p>\n\n\n\n# Hierarchical clustering of correlation matrix<\/em><\/p>\n\n\n\ndistance_matrix = 1 – abs(corr_matrix)<\/em><\/p>\n\n\n\nlinkage_matrix = linkage(squareform(distance_matrix), method=’ward’)<\/em><\/p>\n\n\n\n# Create clustered heatmap<\/em><\/p>\n\n\n\nplt.figure(figsize=(12, 10))<\/em><\/p>\n\n\n\ncluster_map = sns.clustermap(<\/em><\/p>\n\n\n\n corr_matrix, <\/em><\/p>\n\n\n\n row_linkage=linkage_matrix, <\/em><\/p>\n\n\n\n col_linkage=linkage_matrix,<\/em><\/p>\n\n\n\n cmap=’RdBu_r’, <\/em><\/p>\n\n\n\n center=0,<\/em><\/p>\n\n\n\n square=True,<\/em><\/p>\n\n\n\n annot=True,<\/em><\/p>\n\n\n\n fmt=’.2f’<\/em><\/p>\n\n\n\n)<\/em><\/p>\n\n\n\nplt.show()<\/em><\/p>\n\n\n\nAdvanced Python Visual Example – Customer Lifetime Value Distribution:<\/strong><\/p>\n\n\n\nimport matplotlib.pyplot as plt<\/em><\/p>\n\n\n\nimport seaborn as sns<\/em><\/p>\n\n\n\nfrom scipy import stats<\/em><\/p>\n\n\n\nimport numpy as np<\/em><\/p>\n\n\n\n# Calculate CLV if not already in dataset<\/em><\/p>\n\n\n\nif ‘clv’ not in dataset.columns:<\/em><\/p>\n\n\n\n dataset[‘clv’] = dataset[‘avg_order_value’] * dataset[‘purchase_frequency’] * dataset[‘customer_lifespan’]<\/em><\/p>\n\n\n\n# Create multi-panel visualization<\/em><\/p>\n\n\n\nfig, axes = plt.subplots(2, 2, figsize=(15, 12))<\/em><\/p>\n\n\n\n# CLV Distribution<\/em><\/p>\n\n\n\naxes[0,0].hist(dataset[‘clv’], bins=50, alpha=0.7, color=’skyblue’, edgecolor=’black’)<\/em><\/p>\n\n\n\naxes[0,0].axvline(dataset[‘clv’].mean(), color=’red’, linestyle=’–‘, label=f’Mean: ${dataset[“clv”].mean():.2f}’)<\/em><\/p>\n\n\n\naxes[0,0].set_title(‘CLV Distribution’)<\/em><\/p>\n\n\n\naxes[0,0].set_xlabel(‘Customer Lifetime Value ($)’)<\/em><\/p>\n\n\n\naxes[0,0].legend()<\/em><\/p>\n\n\n\n# CLV by Segment<\/em><\/p>\n\n\n\nsns.boxplot(data=dataset, x=’segment_name’, y=’clv’, ax=axes[0,1])<\/em><\/p>\n\n\n\naxes[0,1].set_title(‘CLV by Customer Segment’)<\/em><\/p>\n\n\n\naxes[0,1].tick_params(axis=’x’, rotation=45)<\/em><\/p>\n\n\n\n# CLV vs Recency<\/em><\/p>\n\n\n\naxes[1,0].scatter(dataset[‘days_since_last_purchase’], dataset[‘clv’], alpha=0.6)<\/em><\/p>\n\n\n\naxes[1,0].set_xlabel(‘Days Since Last Purchase’)<\/em><\/p>\n\n\n\naxes[1,0].set_ylabel(‘CLV ($)’)<\/em><\/p>\n\n\n\naxes[1,0].set_title(‘CLV vs Customer Recency’)<\/em><\/p>\n\n\n\n# Probability plot<\/em><\/p>\n\n\n\nstats.probplot(dataset[‘clv’], dist=”norm”, plot=axes[1,1])<\/em><\/p>\n\n\n\naxes[1,1].set_title(‘CLV Normal Probability Plot’)<\/em><\/p>\n\n\n\nplt.tight_layout()<\/em><\/p>\n\n\n\nplt.show()<\/em><\/p>\n\n\n\n