{"id":12461,"date":"2025-07-06T14:06:17","date_gmt":"2025-07-06T08:21:17","guid":{"rendered":"https:\/\/nestnepal.com\/blog\/?p=12461"},"modified":"2025-08-11T13:08:32","modified_gmt":"2025-08-11T07:23:32","slug":"power-bi-streaming-data-master-the-dashboards","status":"publish","type":"post","link":"https:\/\/nestnepal.com\/blog\/power-bi-streaming-data-master-the-dashboards\/","title":{"rendered":"Real-time Dashboard with Power BI and Streaming Data"},"content":{"rendered":"\n<p>Real-time dashboards in <a href=\"https:\/\/nestnepal.com\/microsoft-power-bi-in-nepal\/\">Power BI<\/a> are where things get exciting and, honestly, a bit tricky. You&#8217;re not just dealing with static reports anymore; you&#8217;re building live, breathing dashboards that update as data flows in. Whether it&#8217;s <a href=\"https:\/\/builtin.com\/articles\/iot-sensors\" target=\"_blank\" rel=\"noopener\">IoT sensors<\/a>, web analytics, financial trading data, or social media feeds, getting real-time data into Power BI requires understanding the different streaming options and their trade-offs.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"735\" height=\"898\" data-src=\"https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/streaming-data.jpeg\" alt=\"streaming-data\" class=\"wp-image-12462 lazyload\" data-srcset=\"https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/streaming-data.jpeg 735w, https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/streaming-data-246x300.jpeg 246w, https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/streaming-data-380x464.jpeg 380w, https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/streaming-data-550x672.jpeg 550w\" data-sizes=\"(max-width: 735px) 100vw, 735px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 735px; --smush-placeholder-aspect-ratio: 735\/898;\" \/><\/figure>\n\n\n\n<p>The challenge isn&#8217;t just technical; it&#8217;s also about designing dashboards that remain useful when data is constantly changing. Let&#8217;s dive into the practical approaches that actually work in production environments.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Understanding Power BI&#8217;s Streaming Architecture<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"736\" height=\"557\" data-src=\"https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/Power-BI-2.jpeg\" alt=\"power-bi\" class=\"wp-image-12463 lazyload\" data-srcset=\"https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/Power-BI-2.jpeg 736w, https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/Power-BI-2-300x227.jpeg 300w, https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/Power-BI-2-380x288.jpeg 380w, https:\/\/nestnepal.com\/blog\/wp-content\/uploads\/2025\/07\/Power-BI-2-550x416.jpeg 550w\" data-sizes=\"(max-width: 736px) 100vw, 736px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 736px; --smush-placeholder-aspect-ratio: 736\/557;\" \/><\/figure>\n\n\n\n<p>Power BI offers three main approaches for real-time data, each with distinct characteristics:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td><strong>Method<\/strong><\/td><td><strong>Update Frequency<\/strong><\/td><td><strong>Data Retention<\/strong><\/td><td><strong>Historical Analysis<\/strong><\/td><td><strong>Best For<\/strong><\/td><\/tr><tr><td><strong>Streaming Datasets<\/strong><\/td><td>Real-time<\/td><td>1 hour (200k rows)<\/td><td>Limited<\/td><td>Live metrics, IoT sensors<\/td><\/tr><tr><td><strong>Push Datasets<\/strong><\/td><td>Near real-time<\/td><td>Permanent<\/td><td>Full<\/td><td>Operational dashboards<\/td><\/tr><tr><td><strong>DirectQuery<\/strong><\/td><td>On-demand<\/td><td>Full database<\/td><td>Complete<\/td><td>Large-scale real-time<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>The key is picking the right method for your use case. Need to show live sensor readings? Streaming datasets. Want to track sales performance with historical context? Push datasets. Building an enterprise monitoring dashboard? DirectQuery.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Method 1: Streaming Datasets (True Real-time)<\/strong><\/h2>\n\n\n\n<p>Streaming datasets is Power BI&#8217;s fastest option; data appears on dashboards within seconds of being sent. Perfect for live monitoring scenarios where you need immediate visibility.<\/p>\n\n\n\n<p><strong>Setting up a streaming dataset:<\/strong><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>In Power BI Service, create a new streaming dataset<\/li>\n\n\n\n<li>Define your data schema (columns and types)<\/li>\n\n\n\n<li>Get the REST API endpoint<\/li>\n\n\n\n<li>Start pushing data<\/li>\n<\/ol>\n\n\n\n<p><strong>Creating the dataset schema:<\/strong><\/p>\n\n\n\n<p><em>{<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&#8220;name&#8221;: &#8220;IoT_Sensor_Stream&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&#8220;columns&#8221;: [<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;{&#8220;name&#8221;: &#8220;timestamp&#8221;, &#8220;dataType&#8221;: &#8220;DateTime&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;{&#8220;name&#8221;: &#8220;device_id&#8221;, &#8220;dataType&#8221;: &#8220;String&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;{&#8220;name&#8221;: &#8220;temperature&#8221;, &#8220;dataType&#8221;: &#8220;Number&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;{&#8220;name&#8221;: &#8220;humidity&#8221;, &#8220;dataType&#8221;: &#8220;Number&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;{&#8220;name&#8221;: &#8220;pressure&#8221;, &#8220;dataType&#8221;: &#8220;Number&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;{&#8220;name&#8221;: &#8220;location&#8221;, &#8220;dataType&#8221;: &#8220;String&#8221;}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;]<\/em><\/p>\n\n\n\n<p><em>}<\/em><\/p>\n\n\n\n<p><strong>Python script to push streaming data:<\/strong><\/p>\n\n\n\n<p><em>import requests<\/em><\/p>\n\n\n\n<p><em>import json<\/em><\/p>\n\n\n\n<p><em>import time<\/em><\/p>\n\n\n\n<p><em>import random<\/em><\/p>\n\n\n\n<p><em>from datetime import datetime<\/em><\/p>\n\n\n\n<p><em># Your streaming dataset endpoint from Power BI<\/em><\/p>\n\n\n\n<p><em>POWERBI_ENDPOINT = &#8220;https:\/\/api.powerbi.com\/beta\/[your-workspace]\/datasets\/[dataset-id]\/rows?key=[your-key]&#8221;<\/em><\/p>\n\n\n\n<p><em>def generate_sensor_data():<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Simulate IoT sensor readings&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;devices = [&#8216;sensor_001&#8217;, &#8216;sensor_002&#8217;, &#8216;sensor_003&#8217;, &#8216;sensor_004&#8217;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;locations = [&#8216;Factory_Floor&#8217;, &#8216;Warehouse&#8217;, &#8216;Office&#8217;, &#8216;Shipping_Dock&#8217;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;return {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;timestamp&#8221;: datetime.utcnow().isoformat() + &#8220;Z&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;device_id&#8221;: random.choice(devices),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;temperature&#8221;: round(random.uniform(18.0, 35.0), 2),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;humidity&#8221;: round(random.uniform(30.0, 80.0), 2),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;pressure&#8221;: round(random.uniform(980.0, 1020.0), 2),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;location&#8221;: random.choice(locations)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;}<\/em><\/p>\n\n\n\n<p><em>def push_to_powerbi(data):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Send data to Power BI streaming dataset&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;headers = {&#8216;Content-Type&#8217;: &#8216;application\/json&#8217;}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;# Power BI expects an array of rows<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;payload = [data]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;try:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;response = requests.post(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;POWERBI_ENDPOINT,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;headers=headers,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data=json.dumps(payload),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;timeout=10<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if response.status_code == 200:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2713 Data pushed successfully: {data[&#8216;device_id&#8217;]} &#8211; {data[&#8216;temperature&#8217;]}\u00b0C&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2717 Error: {response.status_code} &#8211; {response.text}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;except requests.exceptions.RequestException as e:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2717 Request failed: {e}&#8221;)<\/em><\/p>\n\n\n\n<p><em># Main streaming loop<\/em><\/p>\n\n\n\n<p><em>def start_streaming():<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;print(&#8220;Starting IoT data stream to Power BI&#8230;&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;while True:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Generate and send sensor data<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sensor_data = generate_sensor_data()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;push_to_powerbi(sensor_data)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Wait before next reading (adjust based on your needs)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;time.sleep(5)&nbsp; # 5-second intervals<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except KeyboardInterrupt:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(&#8220;\\nStopping data stream&#8230;&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;break<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except Exception as e:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;Unexpected error: {e}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;time.sleep(10)&nbsp; # Wait before retrying<\/em><\/p>\n\n\n\n<p><em>if_name_ == &#8220;_main_&#8221;:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;start_streaming()<\/em><\/p>\n\n\n\n<p><strong>Advanced streaming with batch processing:<\/strong><\/p>\n\n\n\n<p><em>import asyncio<\/em><\/p>\n\n\n\n<p><em>import aiohttp<\/em><\/p>\n\n\n\n<p><em>from collections import deque<\/em><\/p>\n\n\n\n<p><em>import json<\/em><\/p>\n\n\n\n<p><em>class PowerBIStreamer:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def_init_(self, endpoint, batch_size=100, flush_interval=10):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.endpoint = endpoint<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.batch_size = batch_size<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.flush_interval = flush_interval<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.buffer = deque()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.last_flush = time.time()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;async def add_data(self, data):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Add data to buffer and flush if needed&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.buffer.append(data)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Flush if buffer is full or time interval reached<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (len(self.buffer) &gt;= self.batch_size or&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;time.time() &#8211; self.last_flush &gt;= self.flush_interval):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;await self.flush_buffer()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;async def flush_buffer(self):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Send buffered data to Power BI&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if not self.buffer:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Convert buffer to list and clear<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;batch = list(self.buffer)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.buffer.clear()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.last_flush = time.time()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;async with aiohttp.ClientSession() as session:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;async with session.post(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.endpoint,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;headers={&#8216;Content-Type&#8217;: &#8216;application\/json&#8217;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data=json.dumps(batch),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;timeout=aiohttp.ClientTimeout(total=30)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;) as response:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if response.status == 200:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2713 Batch of {len(batch)} records sent successfully&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;error_text = await response.text()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2717 Batch failed: {response.status} &#8211; {error_text}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except asyncio.TimeoutError:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(&#8220;\u2717 Timeout sending batch to Power BI&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except Exception as e:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2717 Error sending batch: {e}&#8221;)<\/em><\/p>\n\n\n\n<p><em># Usage example<\/em><\/p>\n\n\n\n<p><em>async def stream_high_volume_data():<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;streamer = PowerBIStreamer(POWERBI_ENDPOINT, batch_size=50, flush_interval=5)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;# Simulate high-frequency data<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;for i in range(1000):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data = generate_sensor_data()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;await streamer.add_data(data)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;await asyncio.sleep(0.1)&nbsp; # 10 records per second<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;# Ensure final flush<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;await streamer.flush_buffer()<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Method 2: Push Datasets (Near Real-time with History)<\/strong><\/h2>\n\n\n\n<p>Push datasets give you the best of both worlds, near real-time updates with full historical data retention. This is usually the sweet spot for business dashboards.<\/p>\n\n\n\n<p><strong>Creating a push dataset with PowerShell:<\/strong><\/p>\n\n\n\n<p><em># Install Power BI PowerShell module<\/em><\/p>\n\n\n\n<p><em>Install-Module -Name MicrosoftPowerBIMgmt<\/em><\/p>\n\n\n\n<p><em># Connect to Power BI<\/em><\/p>\n\n\n\n<p><em>Connect-PowerBIServiceAccount<\/em><\/p>\n\n\n\n<p><em># Define dataset schema<\/em><\/p>\n\n\n\n<p><em>$datasetSchema = @{<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;name = &#8220;Sales_Performance_Push&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;tables = @(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@{<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;name = &#8220;SalesData&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;columns = @(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@{name = &#8220;SaleID&#8221;; dataType = &#8220;String&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@{name = &#8220;Timestamp&#8221;; dataType = &#8220;DateTime&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@{name = &#8220;Amount&#8221;; dataType = &#8220;Double&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@{name = &#8220;Product&#8221;; dataType = &#8220;String&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@{name = &#8220;SalesRep&#8221;; dataType = &#8220;String&#8221;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;@{name = &#8220;Region&#8221;; dataType = &#8220;String&#8221;}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>}<\/em><\/p>\n\n\n\n<p><em># Create a dataset<\/em><\/p>\n\n\n\n<p><em>$dataset = New-PowerBIDataset -Dataset $datasetSchema -WorkspaceId $workspaceId<\/em><\/p>\n\n\n\n<p><strong>Python implementation for push datasets:<\/strong><\/p>\n\n\n\n<p><em>import requests<\/em><\/p>\n\n\n\n<p><em>import json<\/em><\/p>\n\n\n\n<p><em>import pandas as pd<\/em><\/p>\n\n\n\n<p><em>from datetime import datetime, timedelta<\/em><\/p>\n\n\n\n<p><em>import time<\/em><\/p>\n\n\n\n<p><em>class PowerBIPushDataset:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def_init_(self, workspace_id, dataset_id, access_token):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.workspace_id = workspace_id<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.dataset_id = dataset_id<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.access_token = access_token<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.base_url = f&#8221;https:\/\/api.powerbi.com\/v1.0\/myorg\/groups\/{workspace_id}\/datasets\/{dataset_id}&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def get_headers(self):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;Authorization&#8217;: f&#8217;Bearer {self.access_token}&#8217;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;Content-Type&#8217;: &#8216;application\/json&#8217;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def add_rows(self, table_name, rows):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Add rows to a push dataset table&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;url = f&#8221;{self.base_url}\/tables\/{table_name}\/rows&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;payload = {&#8220;rows&#8221;: rows}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;response = requests.post(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;url,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;headers=self.get_headers(),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data=json.dumps(payload)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if response.status_code == 200:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2713 Successfully added {len(rows)} rows to {table_name}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return True<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2717 Error adding rows: {response.status_code} &#8211; {response.text}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return False<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def clear_table(self, table_name):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Clear all rows from a table&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;url = f&#8221;{self.base_url}\/tables\/{table_name}\/rows&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;response = requests.delete(url, headers=self.get_headers())<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if response.status_code == 200:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2713 Table {table_name} cleared successfully&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return True<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2717 Error clearing table: {response.status_code}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return False<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def refresh_dataset(self):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Trigger dataset refresh&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;url = f&#8221;{self.base_url}\/refreshes&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;response = requests.post(url, headers=self.get_headers())<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if response.status_code == 202:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(&#8220;\u2713 Dataset refresh triggered&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return True<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2717 Error triggering refresh: {response.status_code}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return False<\/em><\/p>\n\n\n\n<p><em># Real-world usage example: Sales dashboard<\/em><\/p>\n\n\n\n<p><em>def stream_sales_data():<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;# Initialize push dataset client<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;pbi_client = PowerBIPushDataset(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;workspace_id=&#8221;your-workspace-id&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dataset_id=&#8221;your-dataset-id&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;access_token=&#8221;your-access-token&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;# Simulate sales transactions<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;products = [&#8216;Widget A&#8217;, &#8216;Widget B&#8217;, &#8216;Gadget X&#8217;, &#8216;Gadget Y&#8217;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;sales_reps = [&#8216;John Smith&#8217;, &#8216;Jane Doe&#8217;, &#8216;Mike Johnson&#8217;, &#8216;Sarah Wilson&#8217;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;regions = [&#8216;North&#8217;, &#8216;South&#8217;, &#8216;East&#8217;, &#8216;West&#8217;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;while True:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Generate batch of sales records<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sales_batch = []<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;batch_size = random.randint(5, 20)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for _ in range(batch_size):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sale = {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;SaleID&#8221;: f&#8221;TXN_{int(time.time())}_{random.randint(1000, 9999)}&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Timestamp&#8221;: datetime.utcnow().isoformat() + &#8220;Z&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Amount&#8221;: round(random.uniform(50.0, 5000.0), 2),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Product&#8221;: random.choice(products),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;SalesRep&#8221;: random.choice(sales_reps),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Region&#8221;: random.choice(regions)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sales_batch.append(sale)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Push to Power BI<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;success = pbi_client.add_rows(&#8220;SalesData&#8221;, sales_batch)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if success:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;total_amount = sum(sale[&#8216;Amount&#8217;] for sale in sales_batch)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;Batch total: ${total_amount:.2f}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Wait before next batch<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;time.sleep(random.randint(10, 30))<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except KeyboardInterrupt:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(&#8220;\\nStopping sales data stream&#8230;&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;break<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except Exception as e:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;Error in sales stream: {e}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;time.sleep(60)&nbsp; # Wait before retrying<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Method 3: DirectQuery for Enterprise Real-time<\/strong><\/h2>\n\n\n\n<p>For large-scale enterprise scenarios, DirectQuery against a real-time database often provides the best performance and flexibility.<\/p>\n\n\n\n<p><strong>Setting up real-time DirectQuery:<\/strong><\/p>\n\n\n\n<p><em>&#8212; Example: Creating a real-time view for Power BI<\/em><\/p>\n\n\n\n<p><em>CREATE VIEW vw_RealTimeSales AS<\/em><\/p>\n\n\n\n<p><em>SELECT&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;s.SaleID,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;s.Timestamp,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;s.Amount,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;s.ProductID,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;p.ProductName,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;s.CustomerID,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;c.CustomerName,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;s.RegionID,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;r.RegionName,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8212; Add calculated fields for better performance<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;CASE&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;WHEN s.Amount &gt; 1000 THEN &#8216;High Value&#8217;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;WHEN s.Amount &gt; 500 THEN &#8216;Medium Value&#8217;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ELSE &#8216;Low Value&#8217;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;END as SaleCategory,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8212; Pre-calculate time-based groupings<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;DATEPART(hour, s.Timestamp) as SaleHour,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;CAST(s.Timestamp as DATE) as SaleDate<\/em><\/p>\n\n\n\n<p><em>FROM Sales s<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;JOIN Products p ON s.ProductID = p.ProductID<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;JOIN Customers c ON s.CustomerID = c.CustomerID<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;JOIN Regions r ON s.RegionID = r.RegionID<\/em><\/p>\n\n\n\n<p><em>WHERE s.Timestamp &gt;= DATEADD(day, -7, GETDATE())&nbsp; &#8212; Only recent data for performance<\/em><\/p>\n\n\n\n<p><strong>Optimizing DirectQuery for real-time performance:<\/strong><\/p>\n\n\n\n<p><em>&#8212; Create covering indexes for common Power BI queries<\/em><\/p>\n\n\n\n<p><em>CREATE NONCLUSTERED INDEX IX_Sales_RealTime_Covering<\/em><\/p>\n\n\n\n<p><em>ON Sales (Timestamp DESC, RegionID, ProductID)<\/em><\/p>\n\n\n\n<p><em>INCLUDE (Amount, CustomerID, SaleID)<\/em><\/p>\n\n\n\n<p><em>&#8212; Partitioned view for time-based queries<\/em><\/p>\n\n\n\n<p><em>CREATE VIEW vw_Sales_Last24Hours AS<\/em><\/p>\n\n\n\n<p><em>SELECT * FROM vw_RealTimeSales<\/em><\/p>\n\n\n\n<p><em>WHERE Timestamp &gt;= DATEADD(hour, -24, GETDATE())<\/em><\/p>\n\n\n\n<p><em>&#8212; Aggregate tables for better performance<\/em><\/p>\n\n\n\n<p><em>CREATE VIEW vw_Sales_HourlyAggregates AS<\/em><\/p>\n\n\n\n<p><em>SELECT&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;CAST(Timestamp as DATE) as SaleDate,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;DATEPART(hour, Timestamp) as SaleHour,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;RegionID,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;ProductID,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;COUNT(*) as TransactionCount,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;SUM(Amount) as TotalAmount,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;AVG(Amount) as AverageAmount,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;MAX(Timestamp) as LastUpdated<\/em><\/p>\n\n\n\n<p><em>FROM Sales<\/em><\/p>\n\n\n\n<p><em>WHERE Timestamp &gt;= DATEADD(day, -30, GETDATE())<\/em><\/p>\n\n\n\n<p><em>GROUP BY CAST(Timestamp as DATE), DATEPART(hour, Timestamp), RegionID, ProductID<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Building Effective Real-time Dashboards<\/strong><\/h2>\n\n\n\n<p><strong>Key design principles for real-time dashboards:<\/strong><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Hierarchy of Information:<\/strong> Most important metrics prominently displayed<\/li>\n\n\n\n<li><strong>Visual Stability:<\/strong> Avoid constantly jumping numbers that are hard to read<\/li>\n\n\n\n<li><strong>Context Preservation:<\/strong> Show trends, not just current values<\/li>\n\n\n\n<li><strong>Performance Optimization:<\/strong> Fast loading and smooth updates<\/li>\n<\/ol>\n\n\n\n<p><strong>Essential real-time dashboard components:<\/strong><\/p>\n\n\n\n<p><em>{<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&#8220;dashboard_layout&#8221;: {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8220;top_row&#8221;: {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;kpi_cards&#8221;: [<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Current Revenue&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Active Users&#8221;,&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;System Status&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Alert Count&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8220;middle_section&#8221;: {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;main_chart&#8221;: &#8220;Real-time trend line&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;secondary_charts&#8221;: [<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Geographic distribution&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;Category breakdown&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;},<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8220;bottom_section&#8221;: {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;data_table&#8221;: &#8220;Recent transactions&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;status_indicators&#8221;: &#8220;System health&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;}<\/em><\/p>\n\n\n\n<p><em>}<\/em><\/p>\n\n\n\n<p><strong><em>DAX measures for real-time dashboards:<\/em><\/strong><\/p>\n\n\n\n<p><em>&#8212; Current Hour Sales<\/em><\/p>\n\n\n\n<p><em>Current Hour Sales =&nbsp;<\/em><\/p>\n\n\n\n<p><em>CALCULATE(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;SUM(Sales[Amount]),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;FILTER(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales[Timestamp] &gt;= NOW() &#8211; TIME(1,0,0) &amp;&amp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales[Timestamp] &lt;= NOW()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>)<\/em><\/p>\n\n\n\n<p><em>&#8212; Real-time Growth Rate<\/em><\/p>\n\n\n\n<p><em>Hourly Growth Rate =&nbsp;<\/em><\/p>\n\n\n\n<p><em>VAR CurrentHour = [Current Hour Sales]<\/em><\/p>\n\n\n\n<p><em>VAR PreviousHour =&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;CALCULATE(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;SUM(Sales[Amount]),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;FILTER(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales[Timestamp] &gt;= NOW() &#8211; TIME(2,0,0) &amp;&amp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales[Timestamp] &lt; NOW() &#8211; TIME(1,0,0)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>RETURN<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;IF(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;PreviousHour &gt; 0,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DIVIDE(CurrentHour &#8211; PreviousHour, PreviousHour) * 100,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;BLANK()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>&#8212; Live Status Indicator<\/em><\/p>\n\n\n\n<p><em>System Status =&nbsp;<\/em><\/p>\n\n\n\n<p><em>VAR LastDataPoint = MAX(Sales[Timestamp])<\/em><\/p>\n\n\n\n<p><em>VAR MinutesAgo = DATEDIFF(LastDataPoint, NOW(), MINUTE)<\/em><\/p>\n\n\n\n<p><em>RETURN<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;SWITCH(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;TRUE(),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MinutesAgo &lt;= 5, &#8220;\ud83d\udfe2 Live&#8221;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;MinutesAgo &lt;= 15, &#8220;\ud83d\udfe1 Delayed&#8221;,&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;\ud83d\udd34 Stale&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>&#8212; Moving Average for Smoothing<\/em><\/p>\n\n\n\n<p><em>15-Minute Moving Average =&nbsp;<\/em><\/p>\n\n\n\n<p><em>CALCULATE(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;AVERAGE(Sales[Amount]),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;FILTER(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales[Timestamp] &gt;= NOW() &#8211; TIME(0,15,0) &amp;&amp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales[Timestamp] &lt;= NOW()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>)<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Handling Real-time Data Challenges<\/strong><\/h2>\n\n\n\n<p><strong>Data Quality and Validation:<\/strong><\/p>\n\n\n\n<p><em>class DataValidator:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def_init_(self):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.required_fields = [&#8216;timestamp&#8217;, &#8216;value&#8217;, &#8216;source&#8217;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.numeric_fields = [&#8216;value&#8217;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.max_age_minutes = 60<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def validate_record(self, record):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Validate individual data record&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;errors = []<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Check required fields<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for field in self.required_fields:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if field not in record or record[field] is None:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;errors.append(f&#8221;Missing required field: {field}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Validate data types<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for field in self.numeric_fields:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if field in record:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;float(record[field])<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except (ValueError, TypeError):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;errors.append(f&#8221;Invalid numeric value for {field}: {record[field]}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Check timestamp freshness<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if &#8216;timestamp&#8217; in record:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;record_time = datetime.fromisoformat(record[&#8216;timestamp&#8217;].replace(&#8216;Z&#8217;, &#8216;+00:00&#8217;))<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;age_minutes = (datetime.now(timezone.utc) &#8211; record_time).total_seconds() \/ 60<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if age_minutes &gt; self.max_age_minutes:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;errors.append(f&#8221;Data too old: {age_minutes:.1f} minutes&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except ValueError:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;errors.append(f&#8221;Invalid timestamp format: {record[&#8216;timestamp&#8217;]}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return len(errors) == 0, errors<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def validate_batch(self, records):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Validate batch of records&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;valid_records = []<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;invalid_records = []<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for record in records:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;is_valid, errors = self.validate_record(record)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if is_valid:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;valid_records.append(record)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;invalid_records.append({<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;record&#8217;: record,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;errors&#8217;: errors<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;})<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;Invalid record: {errors}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return valid_records, invalid_records<\/em><\/p>\n\n\n\n<p><strong>Error Handling and Resilience:<\/strong><\/p>\n\n\n\n<p><em>class ResilientStreamer:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def_init_(self, endpoint, max_retries=3, backoff_factor=2):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.endpoint = endpoint<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.max_retries = max_retries<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.backoff_factor = backoff_factor<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.failed_batches = deque(maxlen=100)&nbsp; # Store failed batches for retry<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;async def send_with_retry(self, data):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Send data with exponential backoff retry&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for attempt in range(self.max_retries):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;async with aiohttp.ClientSession() as session:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;async with session.post(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.endpoint,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;json=data,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;timeout=aiohttp.ClientTimeout(total=30)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;) as response:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if response.status == 200:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return True<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;elif response.status == 429:&nbsp; # Rate limited<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;wait_time = 2 ** attempt * self.backoff_factor<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;Rate limited, waiting {wait_time}s&#8230;&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;await asyncio.sleep(wait_time)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;HTTP {response.status}: {await response.text()}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except Exception as e:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;wait_time = 2 ** attempt * self.backoff_factor<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;Attempt {attempt + 1} failed: {e}, retrying in {wait_time}s&#8230;&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;await asyncio.sleep(wait_time)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# All retries failed, store for later retry<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.failed_batches.append({<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;data&#8217;: data,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;timestamp&#8217;: datetime.utcnow(),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;attempts&#8217;: self.max_retries<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;})<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return False<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;async def retry_failed_batches(self):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Retry previously failed batches&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;retry_count = 0<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;while self.failed_batches and retry_count &lt; 10:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;batch = self.failed_batches.popleft()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Don&#8217;t retry batches older than 1 hour<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (datetime.utcnow() &#8211; batch[&#8216;timestamp&#8217;]).seconds &gt; 3600:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;continue<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;success = await self.send_with_retry(batch[&#8216;data&#8217;])<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if success:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f&#8221;\u2713 Successfully retried failed batch&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;retry_count += 1<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Put back in the queue if still failing<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.failed_batches.append(batch)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;break<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Performance Monitoring and Optimization<\/strong><\/h2>\n\n\n\n<p><strong>Real-time dashboard performance metrics:<\/strong><\/p>\n\n\n\n<p><em>&#8212; Dashboard Refresh Performance<\/em><\/p>\n\n\n\n<p><em>Last Refresh Duration =&nbsp;<\/em><\/p>\n\n\n\n<p><em>VAR RefreshLog =&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;TOPN(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;SORT(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;SUMMARIZE(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;Refresh Log&#8217;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;Refresh Log'[Timestamp],<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;Refresh Log'[Duration]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;Refresh Log'[Timestamp],<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DESC<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;Refresh Log'[Timestamp],<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DESC<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>RETURN<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;MAXX(RefreshLog, &#8216;Refresh Log'[Duration])<\/em><\/p>\n\n\n\n<p><em>&#8212; Data Freshness Indicator<\/em><\/p>\n\n\n\n<p><em>Data Freshness =&nbsp;<\/em><\/p>\n\n\n\n<p><em>VAR LastUpdate = MAX(Sales[Timestamp])<\/em><\/p>\n\n\n\n<p><em>VAR MinutesOld = DATEDIFF(LastUpdate, NOW(), MINUTE)<\/em><\/p>\n\n\n\n<p><em>RETURN<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;CONCATENATE(&#8220;Last updated: &#8220;,&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CONCATENATE(MinutesOld, &#8221; minutes ago&#8221;))<\/em><\/p>\n\n\n\n<p><em>&#8212; Streaming Performance Health<\/em><\/p>\n\n\n\n<p><em>Stream Health Score =&nbsp;<\/em><\/p>\n\n\n\n<p><em>VAR ExpectedRecordsPerMinute = 50<\/em><\/p>\n\n\n\n<p><em>VAR ActualRecordsLastMinute =&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;CALCULATE(<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;COUNT(Sales[SaleID]),<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sales[Timestamp] &gt;= NOW() &#8211; TIME(0,1,0)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;)<\/em><\/p>\n\n\n\n<p><em>VAR HealthPercentage =&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;DIVIDE(ActualRecordsLastMinute, ExpectedRecordsPerMinute) * 100<\/em><\/p>\n\n\n\n<p><em>RETURN<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;MIN(HealthPercentage, 100)<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Advanced Real-time Scenarios<\/strong><\/h2>\n\n\n\n<p><strong>Multi-source data integration:<\/strong><\/p>\n\n\n\n<p><em>class MultiSourceStreamer:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def _init_(self):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.sources = {}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.merged_buffer = deque(maxlen=1000)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def register_source(self, source_name, schema_mapping):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Register a data source with its schema mapping&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.sources[source_name] = {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;mapping&#8217;: schema_mapping,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;last_seen&#8217;: None,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8216;record_count&#8217;: 0<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def normalize_record(self, source_name, raw_record):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Normalize record from specific source to common schema&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if source_name not in self.sources:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;raise ValueError(f&#8221;Unknown source: {source_name}&#8221;)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mapping = self.sources[source_name][&#8216;mapping&#8217;]<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;normalized = {}<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for target_field, source_field in mapping.items():<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if callable(source_field):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;normalized[target_field] = source_field(raw_record)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;normalized[target_field] = raw_record.get(source_field)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;normalized[&#8216;source&#8217;] = source_name<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;normalized[&#8216;processed_at&#8217;] = datetime.utcnow().isoformat() + &#8220;Z&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return normalized<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;def merge_and_send(self, records):<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;&#8221;&#8221;Merge records from multiple sources and send to Power BI&#8221;&#8221;&#8221;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Sort by timestamp for proper ordering<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sorted_records = sorted(records, key=lambda x: x.get(&#8216;timestamp&#8217;, &#8221;))<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Add to merged buffer<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.merged_buffer.extend(sorted_records)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Send batch if buffer is large enough<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if len(self.merged_buffer) &gt;= 50:<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;batch = list(self.merged_buffer)<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.merged_buffer.clear()<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Send to Power BI<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return self.send_to_powerbi(batch)<\/em><\/p>\n\n\n\n<p><em># Usage example<\/em><\/p>\n\n\n\n<p><em>streamer = MultiSourceStreamer()<\/em><\/p>\n\n\n\n<p><em># Register different data sources<\/em><\/p>\n\n\n\n<p><em>streamer.register_source(&#8216;web_analytics&#8217;, {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8216;timestamp&#8217;: &#8216;event_time&#8217;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8216;user_id&#8217;: &#8216;visitor_id&#8217;,&nbsp;<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8216;value&#8217;: lambda x: x[&#8216;page_views&#8217;] * x[&#8216;session_duration&#8217;]<\/em><\/p>\n\n\n\n<p><em>})<\/em><\/p>\n\n\n\n<p><em>streamer.register_source(&#8216;sales_system&#8217;, {<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8216;timestamp&#8217;: &#8216;transaction_time&#8217;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8216;user_id&#8217;: &#8216;customer_id&#8217;,<\/em><\/p>\n\n\n\n<p><em>&nbsp;&nbsp;&nbsp;&nbsp;&#8216;value&#8217;: &#8216;amount&#8217;<\/em><\/p>\n\n\n\n<p><em>})<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Production Deployment Checklist<\/strong><\/h2>\n\n\n\n<p><strong>Infrastructure considerations:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u2705 Network bandwidth sufficient for streaming volume<\/li>\n\n\n\n<li>\u2705 Power BI Premium capacity is sized appropriately<\/li>\n\n\n\n<li>\u2705 Database connection pooling is configured<\/li>\n\n\n\n<li>\u2705 Monitoring and alerting systems in place<\/li>\n\n\n\n<li>\u2705 Backup streaming endpoints configured<\/li>\n\n\n\n<li>\u2705 Data retention policies defined<\/li>\n<\/ul>\n\n\n\n<p><strong>Security and compliance:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u2705 API keys secured and rotated regularly<\/li>\n\n\n\n<li>\u2705 Data encryption in transit and at rest<\/li>\n\n\n\n<li>\u2705 Access controls and authentication are configured<\/li>\n\n\n\n<li>\u2705 Audit logging enabled<\/li>\n\n\n\n<li>\u2705 Data privacy requirements addressed<\/li>\n<\/ul>\n\n\n\n<p>Real-time dashboards in Power BI can transform how organizations monitor and respond to their business. The key is choosing the right streaming method for your needs, building resilient data pipelines, and designing dashboards that provide actionable insights without overwhelming users. Start with simple streaming scenarios and gradually add complexity as your team gains experience with real-time data flows.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Real-time dashboards in Power BI are where things get exciting and, honestly, a bit tricky. You&#8217;re not just dealing with&#8230;<\/p>\n","protected":false},"author":15,"featured_media":12940,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[422],"tags":[446,445],"class_list":["post-12461","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-microsoft","tag-microsoft","tag-power-bi"],"_links":{"self":[{"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/posts\/12461","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/comments?post=12461"}],"version-history":[{"count":1,"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/posts\/12461\/revisions"}],"predecessor-version":[{"id":12464,"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/posts\/12461\/revisions\/12464"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/media\/12940"}],"wp:attachment":[{"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/media?parent=12461"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/categories?post=12461"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nestnepal.com\/blog\/wp-json\/wp\/v2\/tags?post=12461"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}