<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[GeekInData]]></title><description><![CDATA[GeekInData]]></description><link>https://geekindata.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 16 Nov 2026 20:14:16 GMT</lastBuildDate><atom:link href="https://geekindata.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Fabric Mirroring Lag: Where the Minutes Go]]></title><description><![CDATA[I used AI tools to help draft and edit this post. I checked the measurements and the wording, and the original lives at geekindata.com. The diagrams are on…]]></description><link>https://geekindata.hashnode.dev/fabric-mirroring-lag-where-the-minutes-go</link><guid isPermaLink="true">https://geekindata.hashnode.dev/fabric-mirroring-lag-where-the-minutes-go</guid><category><![CDATA[microsoft fabric]]></category><category><![CDATA[data-engineering]]></category><category><![CDATA[SQL]]></category><category><![CDATA[mirroring]]></category><dc:creator><![CDATA[Sai Prudhvi Neelakantam]]></dc:creator><pubDate>Mon, 16 Nov 2026 09:26:28 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af4b0facb31eae609ac4299/c2ac1424-ad6b-455d-b55a-9d14478be005.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I used AI tools to help draft and edit this post. I checked the measurements and the wording, and the original lives at <a href="https://www.geekindata.com/blog/53-fabric-mirroring-lag-where-it-goes">geekindata.com</a>. The diagrams are on that page.</p>
<p>Changes reached Delta in a Fabric SQL database mirror in 18 seconds, then took about 2 minutes to show on the SQL analytics endpoint. What to promise.</p>
<h2>Key takeaways</h2>
<ul>
<li>In this test, a change committed in a Fabric SQL database left the source in 9.7 seconds median, 15.1 seconds at most (change feed sample, last 3 runs). It became visible on the SQL analytics endpoint after a median of 100 to 158 seconds, and 296 seconds in the worst case.</li>
<li>Mirroring itself was quick: it committed to Delta about every 15 seconds, and a change reached Delta in a median of 18.5 seconds (max 28.9 seconds).</li>
<li>In this setup (a Fabric SQL database on a 64-CU trial, at the rates tested), the bottleneck was the SQL analytics endpoint picking up new Delta commits, in 15 batches 151 to 329 seconds apart. Delta to endpoint was most of the lag: a median 86% of each marker's lag (in medians, 116 s of about 134 s).</li>
<li>Promise "about 2 minutes typical, within 5 minutes" for queries through the SQL analytics endpoint, and only after you have measured your own setup with a heartbeat row.</li>
<li>Do not tune the source to fix this. Capture and the Delta commit were never the slow part.</li>
<li>If someone needs seconds, or needs to act on each change, mirroring plus the endpoint is the wrong path. Put the event on a stream where it is produced.</li>
<li>All numbers were measured on a 64-CU Fabric trial, not a statement about paid F64 or production SLAs.</li>
</ul>
<p>Every Fabric SQL database mirrors itself into OneLake automatically, with no settings to configure, and you get a SQL analytics endpoint over that copy (<a href="https://learn.microsoft.com/en-us/fabric/database/sql/mirroring-overview">SQL database mirroring overview</a>). The first question a stakeholder asks is simple: if I change a row in the app, how soon can I see it in a report? Microsoft's FAQ answers "near real-time latency" and publishes no interval (<a href="https://learn.microsoft.com/en-us/fabric/database/sql/mirroring-faq">mirroring FAQ</a>). That is not a number you can put in a requirement.</p>
<p>So I measured it: a Fabric SQL database's automatic mirroring to OneLake, read through its SQL analytics endpoint. This is the first post in a series of experiments I am running on a Fabric trial capacity, each with its method and numbers. The short answer: capture and the Delta commit take seconds, and the minutes are the endpoint catching up.</p>
<p><strong>Named model: capture is fast, the wait is the endpoint</strong></p>
<ul>
<li>Commit: your transaction ends in the Fabric SQL database.</li>
<li>Capture: the change feed scans the log and publishes the change to a landing zone in OneLake. Seconds.</li>
<li>Delta commit: the change is converted to Delta in OneLake. Mirroring committed about every 15 seconds; a change was in Delta 7.4 to 28.9 seconds after commit.</li>
<li>Endpoint sync: the SQL analytics endpoint picks up the new Delta commits in batches. Minutes.</li>
<li>Query: a T-SQL query on the endpoint can see the row.</li>
</ul>
<p><img src="https://www.geekindata.com/assets/blogs/53-fabric-mirroring-lag-where-it-goes/capture-fast-pickup-slow.svg" alt="Four cards in a row: commit in the source SQL database at time zero; capture and Delta commit, with the change feed scan at a median of 9.7 seconds in the DMV sample for runs 7 to 9 and the change in Delta at a median of 18.5 seconds, max 28.9 seconds, across all 153 markers; endpoint sync, highlighted, a median of 116 seconds from Delta to visible, in batches every 151 to 329 seconds, sync type not confirmed; and query on the SQL analytics endpoint with a median of 100 to 158 seconds and p95 of 212 to 276 seconds by nearest rank. A short bar marks the first two stages as seconds and a long bar marks the endpoint sync as minutes, about 86 percent of the lag for the median marker." /></p>
<p><em>Figure 1. The change is in Delta within seconds. The minutes a user feels are the SQL analytics endpoint picking it up.</em></p>
<h2>What I measured, and how</h2>
<p>The question: how long does a committed change in a Fabric SQL database take to become queryable on its own SQL analytics endpoint, at about 10, 100 and 1,000 changed rows per second? I measured at the SQL analytics endpoint, not at the files in OneLake.</p>
<ul>
<li>Setup: one Fabric SQL database with one neutral table, seeded with 1,000,000 deterministic rows, on a 64-CU Fabric trial capacity in one region, on one evening.</li>
<li>Workload: 9 runs of 150 seconds each, in 3 rounds of 10, 100 and 1,000 rows per second, always in that order. One transaction per second, with 40% inserts, 40% updates and 20% deletes, and 60 seconds idle between runs. Achieved rates were 10.1, 100.6 to 100.7 and 1,006.4 to 1,006.5 rows per second, with no late ticks.</li>
<li>Heartbeat markers: every 10 seconds, plus once at the end of each burst, a marker row went in with its commit time from the source clock. That is 17 markers per run, 51 per rate, 153 in total.</li>
<li>Polling: a second thread queried the endpoint every 2 seconds for new markers, with a unique predicate on every poll so no result could come from a cache.</li>
<li>Lag: the first poll that sees a marker, minus its commit time, corrected for the clock offset between the two servers (under 4 ms). Resolution is about 2 seconds.</li>
<li>Capture: I saved the last 30 sessions of <code>sys.dm_change_feed_log_scan_sessions</code> and the contents of <code>sys.dm_change_feed_errors</code> after the runs. That sample covers runs 7 to 9 only. Its latency column is per scan session: the time between the session's end and the commit it was processing. To time each marker, I matched it to the first scan session that processed past its commit.</li>
<li>Delta commits: after the runs I read the mirrored table's Delta log in OneLake, read only: 102 commits over the night. The replicator writes no commitInfo timestamp, so a commit's time is the lastModified time of its log file in OneLake, a close proxy with 1 second resolution. Each marker was matched to the first commit whose newest source change reaches the marker's commit time.</li>
</ul>
<p>All SQL ran from Fabric Python notebooks. No number below is typed by hand; they come from the run logs.</p>
<p><img src="https://www.geekindata.com/assets/blogs/53-fabric-mirroring-lag-where-it-goes/endpoint-marker-age-query.png" alt="SQL analytics endpoint query results for seven heartbeat marker rows in a synthetic table, with the columns id, segment, attribute_a, committed_utc, now_utc and age_s. The markers were committed 10 seconds apart, and age_s reads 116 for the newest and 126, 136, 146, 156, 166 and 176 for the older ones." /></p>
<p><em>Figure 2. Each heartbeat marker carries its commit time from the source, and the query compares it to the endpoint's clock. The lag in this post is that difference at the first poll that sees the marker.</em></p>
<h2>Results</h2>
<p>All 153 markers were seen. Over the 51 markers per rate:</p>
<table>
<thead>
<tr>
<th>Changed rows per second</th>
<th>Median lag</th>
<th>p95 lag (nearest rank)</th>
<th>Min</th>
<th>Max</th>
</tr>
</thead>
<tbody><tr>
<td>10</td>
<td>158 s</td>
<td>276 s</td>
<td>32 s</td>
<td>296 s</td>
</tr>
<tr>
<td>100</td>
<td>130 s</td>
<td>244 s</td>
<td>18 s</td>
<td>264 s</td>
</tr>
<tr>
<td>1,000</td>
<td>100 s</td>
<td>212 s</td>
<td>20 s</td>
<td>214 s</td>
</tr>
</tbody></table>
<p>p95 is nearest rank: the 49th of 51 sorted values, rank ceil(0.95 x n). Across all 153 markers, 16% were visible within 60 seconds, 44% within 2 minutes, 75% within 3 minutes, 94% within 4 minutes and all of them within 5 minutes.</p>
<p><img src="https://www.geekindata.com/assets/blogs/53-fabric-mirroring-lag-where-it-goes/lag-per-rate-and-marker.svg" alt="Left: per change rate, a bar from median to p95 by nearest rank with min and max dots; medians 158, 130 and 100 seconds and p95 276, 244 and 212 seconds for 10, 100 and 1,000 rows per second. Right: every marker's lag plotted against commit time over about 50 minutes, forming falling diagonal runs that reset upward, a sawtooth." /></p>
<p><em>Figure 3. The sawtooth on the right is the finding. Markers committed early in a cycle wait longest; markers committed just before a batch wait only seconds.</em></p>
<p>More numbers from the same night:</p>
<ul>
<li>Capture: for the 41 markers in the DMV sample (runs 7 to 9), the change left the source 9.7 seconds after commit at the median, and 15.1 seconds at most. The time from there to visible on the endpoint was 110 seconds at the median (12 to 171 seconds), about 92% of each marker's lag. Every scan session in the sample finished with zero errors and nothing left to publish, and the errors DMV was empty.</li>
<li>Delta commit: across all 153 markers, a change was in Delta 7.4 to 28.9 seconds after commit (median 18.5 seconds, p95 27.3 seconds). From that Delta commit to visible on the endpoint took 2.8 to 278.6 seconds (median 116.3 seconds, p95 218.6 seconds).</li>
<li>Initial snapshot: the 1,000,000-row seed took 5.5 seconds to insert, reached Delta about 46 seconds after the commit, and the full count was visible on the endpoint 125 seconds after the commit. Until then, the endpoint returned "Invalid object name" for the new table, so that time includes the endpoint discovering a brand new table.</li>
<li>Count convergence: after each burst, source and endpoint row counts matched after 88 to 216 seconds.</li>
<li>Run to run: run medians for the same setup ranged from 98 to 218 seconds.</li>
</ul>
<h2>Where the lag actually comes from</h2>
<p>Capture took seconds, and so did the Delta commit. During every burst, mirroring committed a Delta Merge every 14 to 16 seconds (median 15.0 seconds). So the batching is not in replication. It sits between the Delta table and the SQL analytics endpoint.</p>
<p><img src="https://www.geekindata.com/assets/blogs/53-fabric-mirroring-lag-where-it-goes/replication-monitor-running.png" alt="Replication monitor pane for the Fabric SQL database, with the database name and ID blurred. Status is Running, and the table list shows one table, dbo.table_a, Running, with 1,532,114 rows replicated and a last completed time of 10:09:36 PM GMT+1." /></p>
<p><em>Figure 4. The replication monitor said Running, with a recent last completed time. It reports on replication into OneLake, not on when the SQL analytics endpoint can see the row, and that second step was most of the lag here (in medians, 116 s of about 134 s).</em></p>
<p>Figure 3 shows how: rows did not trickle in. Across the night there were 15 batches of newly visible markers, 151 to 329 seconds apart (median 188 seconds). Each batch picked up between 1 and 11 Delta commits at once, and every marker waiting at that moment appeared together.</p>
<p>That one fact explains the spread in the table:</p>
<ul>
<li>A marker committed just before a batch shows up in about 18 to 32 seconds. That is the min column.</li>
<li>A marker committed just after a batch waits for the next one, up to almost 5 minutes. That is the max column.</li>
<li>The median sits around the middle of the cycle.</li>
</ul>
<p>For context, Learn says that under normal operating conditions the lag between a lakehouse and its SQL analytics endpoint is less than one minute, varying "from a few seconds to minutes" (<a href="https://learn.microsoft.com/en-us/fabric/data-warehouse/sql-analytics-endpoint-performance">SQL analytics endpoint performance</a>). Here, Delta to endpoint took a median of 116 seconds, well above that. That is one evening on one trial, so I read it as what this endpoint did that night, not as a general figure.</p>
<p>Which metadata sync was running is not confirmed. On 30 Oct 2026 the endpoint had the objects Learn lists for the new metadata sync (preview), <code>sys.dm_db_external_tables_log_status</code> and <code>sys.sp_dw_refresh_ext_table</code>, but that DMV returned no rows, and the portal showed no metadata sync (preview) setting. That is consistent with the legacy metadata sync, but it is not proof. Learn says the new metadata sync keeps data queryable "within seconds" of landing and applies only to endpoints created after it is enabled (<a href="https://learn.microsoft.com/en-us/fabric/data-engineering/sql-analytics-endpoint-metadata-sync">metadata sync</a>).</p>
<p>On the change rate: the median was lower at higher rates in every round. I am not claiming the rate caused that. Inside every round the order was always 10, then 100, then 1,000 rows per second, and the first round came right after the initial snapshot, so rate and position in the round cannot be separated. Each run's 17 markers also fell into only 1 to 3 batches, so the real sample is the number of batches, not 51 markers per rate. Batching explains the spread; it does not explain a trend by rate.</p>
<h2>What to promise stakeholders</h2>
<p>Write the promise as a number, about the query path people will actually use:</p>
<ul>
<li>For reports and ad hoc T-SQL on the SQL analytics endpoint: "about 2 minutes typical, within 5 minutes". That covered every marker in this test, with a little margin over the 296-second worst case.</li>
<li>Do not promise "near real-time". That is Microsoft's wording, and people hear seconds. Learn's own endpoint page says minutes are possible.</li>
<li>Do not promise anything for a quiet system yet. Learn says the endpoint's background sync process "stops after 15 minutes without query activity". That is written for lakehouses; the SQL database's endpoint is the same lakehouse-style endpoint, and a Fabric SQL database can pause when idle, which also pauses pending mirroring. My 2-second polling kept the endpoint busy, and I built a probe for the first change after an idle spell but did not run it. The cold case is not measured here.</li>
<li>Re-measure on your own capacity. This is one table, one region, one evening on a trial, with a heavy TPC-H-derived build on the same capacity at the time. Your cycle may be shorter or longer.</li>
</ul>
<p><img src="https://www.geekindata.com/assets/blogs/53-fabric-mirroring-lag-where-it-goes/freshness-promise-paths.svg" alt="Three lanes. Minutes are fine, for refresh-style reports, reconciliation and next-hour decisions: mirroring plus SQL analytics endpoint, what this test measured. Under a minute, for analytics that should feel live with no action per row: read the Delta tables directly, not measured here, test first. Seconds or act per change, for alerts, operational screens and a trigger per event: stream the event with Eventstream, do not poll a mirror for this." /></p>
<p><em>Figure 5. Agree on the freshness number first. The number picks the path, not the other way round.</em></p>
<h2>When mirroring is fine, and when you need a stream</h2>
<p>Mirroring plus the SQL analytics endpoint is a good default when minutes are acceptable: reporting over operational data, reconciliation, and analysis that does not trigger an action per row. There is no pipeline to build, and Microsoft's FAQ states there is no compute cost for mirroring from a Fabric SQL database to OneLake, and mirroring storage is free up to a limit based on capacity. Your queries on the endpoint are billed as normal. Capacity use for this test is not measured yet; when it lands, it will be the cost of the test harness (endpoint queries and notebooks), not replication.</p>
<p>It is the wrong tool when someone needs to react within seconds: an alert, an operational screen, a workflow that fires per change. Polling a mirror every few seconds cannot beat a cycle measured in minutes. Put the event on a stream where it is produced, with Eventstream into an Eventhouse or Activator, and keep mirroring for the analytical copy.</p>
<p>In between sits "should feel live, but nobody acts per row". Engines that read the Delta tables in OneLake directly, such as Spark or a Direct Lake model, skip the SQL analytics endpoint sync. Since every change here was in Delta within about 30 seconds, they may see changes much sooner than the endpoint (Direct Lake once it reframes, for example with automatic updates on). I did not test that path, so test it before you promise it.</p>
<h2>How to watch it</h2>
<p>These are the monitoring routes that apply to a Fabric SQL database's mirror:</p>
<ul>
<li>Portal: on the SQL database, the Replication tab, then Monitor replication, shows the current status, rows replicated and last refresh.</li>
<li>Source DMVs: <code>sys.dm_change_feed_log_scan_sessions</code> for capture timing, and <code>sys.dm_change_feed_errors</code> for failures.</li>
<li>Endpoint side: the Refresh SQL endpoint metadata REST call returns a <code>lastSuccessfulSyncDateTime</code> per table. It also triggers a sync, so treat it as an action, not a passive read.</li>
<li>A heartbeat row: the only one that measures what a user feels. See below.</li>
</ul>
<p>The <code>getMirroringStatus</code> REST call does not apply here. It belongs to mirrored database items, while the SQL database API only offers Start and Stop for mirroring. A 404 for a SQL database is expected.</p>
<h2>Common mistakes</h2>
<ul>
<li>Treating "near real-time" as a number. Microsoft publishes no interval for this. Measure your own.</li>
<li>Tuning the source. Faster capture does not help when the delay is the endpoint picking up Delta commits.</li>
<li>Measuring with row counts only. Counts converge eventually and hide the per-row spread. A timestamped heartbeat row shows the sawtooth.</li>
<li>Letting a cache answer the probe. If every poll is the same query, a cached result can make freshness look worse or better than it is. Vary the predicate.</li>
<li>Reporting the best run, or the p95 as the worst case. Run medians in this test ranged from 98 to 218 seconds for the same setup. Report all runs, the p95 with its method, and the max as the worst case.</li>
<li>Reading a trend from a fixed order. If the rates always run in the same order, any trend by rate is also a trend by position. Shuffle the order per round.</li>
</ul>
<h2>Use as-is or adapt</h2>
<p>A heartbeat table is the cheapest way to know your real lag, and to alert when it drifts. On the Fabric SQL database:</p>
<pre><code class="language-sql">-- datetime2(6): mirrored Delta and the SQL analytics endpoint keep six fractional digits; (7) would be trimmed
CREATE TABLE dbo.heartbeat (
    id            bigint       NOT NULL PRIMARY KEY,
    committed_utc datetime2(6) NOT NULL
);

-- run on a schedule, for example every minute
INSERT INTO dbo.heartbeat (id, committed_utc)
SELECT ISNULL(MAX(id), 0) + 1, SYSUTCDATETIME() FROM dbo.heartbeat;
</code></pre>
<p>On the SQL analytics endpoint:</p>
<pre><code class="language-sql">SELECT MAX(committed_utc) AS last_seen_commit_utc,
       DATEDIFF(second, MAX(committed_utc), SYSUTCDATETIME()) AS staleness_s
FROM dbo.heartbeat;
</code></pre>
<p>If <code>staleness_s</code> goes above the number you promised plus the heartbeat interval (60 s here), you know before the stakeholder does. The six-digit note comes from the Delta precision limit in the mirroring limitations and the data types page. Two clocks are involved, source and endpoint; in my runs the offset was under 4 ms, but check yours before you trust single-digit seconds.</p>
<h2>Method notes and caveats</h2>
<ul>
<li>Measured on a 64-CU Fabric trial, not a statement about paid F64 or production: 153 markers, 9 runs, one evening, one region, one database, one table.</li>
<li>A TPC-H-derived build ran on the same trial capacity during the runs, so noisy neighbour effects cannot be ruled out.</li>
<li>The capture numbers come from the DMV sample for runs 7 to 9 (41 markers), not the whole night. The Delta numbers cover all 153 markers.</li>
<li>Delta commit times are the log files' lastModified time in OneLake (1 second resolution), because the replicator writes no commitInfo timestamp.</li>
<li>Each run's p95 equals its max at 17 markers, so only the pooled p95 per rate is reported.</li>
<li>Sync type: not confirmed.</li>
<li>3 runs per rate, always in the same order, so no claim about rate.</li>
<li>p95 is nearest rank, ceil(0.95 x n). The max is the worst case.</li>
<li>Capacity use (CU): not measured yet. A later figure will cover the test harness (endpoint queries and notebooks), not replication.</li>
<li>The idle (cold) case was not run.</li>
<li>Lag is end to end through the SQL analytics endpoint. Readers of the Delta table directly may see changes much sooner, untested here.</li>
</ul>
<h2>Try it yourself</h2>
<p>The mirroring lag demo in my public tech-demos repo seeds a Fabric SQL database, runs the 10, 100 and 1,000 rows per second workload with heartbeat markers, and polls the SQL analytics endpoint, so you can measure the lag on your own capacity.</p>
]]></content:encoded></item><item><title><![CDATA[Fabric or Databricks? How I Decide]]></title><description><![CDATA[I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that…]]></description><link>https://geekindata.hashnode.dev/fabric-or-databricks-how-i-decide</link><guid isPermaLink="true">https://geekindata.hashnode.dev/fabric-or-databricks-how-i-decide</guid><category><![CDATA[data-engineering]]></category><category><![CDATA[Power BI]]></category><category><![CDATA[microsoft fabric]]></category><category><![CDATA[Databricks]]></category><dc:creator><![CDATA[Sai Prudhvi Neelakantam]]></dc:creator><pubDate>Fri, 13 Nov 2026 09:00:49 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af4b0facb31eae609ac4299/ba972bf7-a433-4dbc-8145-e1f15bd6d773.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at <a href="https://www.geekindata.com/blog/51-fabric-or-databricks-decision">geekindata.com</a>. The diagrams are on that page.</p>
<p>Not a feature war. Five questions I ask before choosing Fabric, Databricks or both, where each choice bites a year later, and when both is honest.</p>
<h2>Key takeaways</h2>
<ul>
<li>Both platforms can run a medallion lakehouse. Feature lists rarely decide; people, consumers, governance and the billing model do.</li>
<li>If Power BI is the main consumer and the team thinks in SQL and reports, Fabric starts ahead. If the team ships code with tests and CI, Databricks does.</li>
<li>A Fabric capacity is shared. A heavy background job can push report users into delays and rejections.</li>
<li>Databricks scales to zero, but someone has to own compute policies, or the bill spreads across many small decisions.</li>
<li>"Both" works when each table has one owner and data flows one way. A mirrored catalog does not carry Unity Catalog grants.</li>
</ul>
<p>Clients rarely ask which platform has more features. They ask which one they will regret less. The honest answer depends less on the platforms than on the people who will run the thing in its second year.</p>
<p>I deliver on both. The medallion patterns I use, SCD2 history in bronze, current-row silver, dimensional gold, look the same on each. What changes is everything around them: who writes the code, how it deploys, who governs access, and how the cost behaves.</p>
<p><strong>Named model: five questions, in this order</strong></p>
<ol>
<li>People: who builds and runs it in year two?</li>
<li>Consumers: what reads the gold layer, and is that Power BI?</li>
<li>Footprint: where does the data already land, and what is already governed?</li>
<li>Cost: a shared capacity you reserve, or compute you pay for per use?</li>
<li>Edge: is there streaming or machine learning that sets the bar?</li>
</ol>
<p><img src="https://www.geekindata.com/assets/blogs/51-fabric-or-databricks-decision/five-questions-tilt.svg" alt="Five questions, each tilting toward Fabric or Databricks: a BI-first team versus an engineering team, Power BI as main consumer versus many consumers, sources landing in Microsoft services versus an existing Unity Catalog setup, steady predictable load versus spiky jobs, report-led real-time versus code-first machine learning" /></p>
<p><em>Figure 1. Each question tilts; none decides alone. Two strong tilts in opposite directions is the signal for both.</em></p>
<h2>Context and constraint</h2>
<p>This is a framework, not a scorecard. I do not weight the questions or add them up; I look for the one or two that are not negotiable for this client, and let those lead. Product status below is as of October 2026, with Learn links where it matters.</p>
<h2>Design</h2>
<h3>1. People: who runs it in year two</h3>
<p>Fabric lowers the entry bar: pipelines, Dataflows, notebooks and Power BI in one workspace and one place to look. That is a real advantage for a team of BI developers and analysts. The bite comes later: low-code workspaces grow fast and deploy badly unless someone brings Git discipline early.</p>
<p>Databricks assumes engineers: code in a repo, bundles, tests, a pipeline. Hand that to a team without engineers and it does not get simpler, it gets abandoned, and notebooks get edited in the workspace.</p>
<p>So my first question is not about data at all: who on the client side can review a pull request?</p>
<h3>2. Consumers: is Power BI the point?</h3>
<p>In most of my projects, Power BI is where the business meets the data, whichever platform sits underneath. If it is the main consumer, Fabric removes a hop: Direct Lake reads Delta tables in OneLake without an import refresh, and the semantic model lives next to the data.</p>
<p>Databricks data can still reach Direct Lake. A <a href="https://learn.microsoft.com/en-us/fabric/mirroring/azure-databricks">mirrored Azure Databricks catalog</a> in Fabric is generally available: Fabric mirrors the catalog structure, reads the tables through shortcuts without copying them, and you can build Direct Lake reports on it. Changes can take seconds to minutes to appear. Plan for two prerequisites: the <code>EXTERNAL USE SCHEMA</code> privilege on the Unity Catalog side, and network access from Fabric to the storage account behind the catalog (<a href="https://learn.microsoft.com/en-us/fabric/mirroring/azure-databricks-tutorial">tutorial</a>). Publishing a catalog to OneLake from the Databricks side is a second route, still in Public Preview (<a href="https://learn.microsoft.com/en-us/azure/databricks/partners/bi/fabric">Fabric with Azure Databricks</a>).</p>
<p>If the gold layer also feeds applications, APIs, other teams' notebooks and models, Power BI is one consumer among many and stops being the deciding factor.</p>
<h3>3. Footprint: what is already governed</h3>
<p>Where the data lands today matters more than where it could land. A client with an existing Unity Catalog setup, groups and grants already in place, should not rebuild that governance in another product to save a hop for reports. A client whose sources already land in Microsoft services, with no lakehouse yet, has nothing to migrate away from.</p>
<p>The bite is governance in two places. Unity Catalog grants do not travel with a mirrored catalog: the same tutorial tells you to set access in Fabric's own permission model, and tables with row filters or column masks are not mirrored at all (<a href="https://learn.microsoft.com/en-us/fabric/mirroring/azure-databricks-limitations">limitations</a>). On the Fabric side, OneLake security is generally available, but its roles do not restrict workspace Admins, Members and Contributors (<a href="https://learn.microsoft.com/en-us/fabric/onelake/security/table-column-row-security">OneLake row and column security</a>). Workspace roles are a security decision, not a convenience.</p>
<p><img src="https://www.geekindata.com/assets/blogs/51-fabric-or-databricks-decision/onelake-role-data.png" alt="OneLake security role demoreaders on a lakehouse: Read permission, type Grant, with the dbo schema as the data in the role" /></p>
<p><em>Figure 2. The OneLake role grants Read on one schema, and it is set on the lakehouse rather than the workspace.</em></p>
<p><img src="https://www.geekindata.com/assets/blogs/51-fabric-or-databricks-decision/onelake-role-members.png" alt="Members in role tab of the demoreaders OneLake role, with one user added by email and the name masked" /></p>
<p><em>Figure 3. Members are added to the OneLake role separately from workspace access, and workspace Admins, Members and Contributors are not restricted by it.</em></p>
<h3>4. Cost: reserved and shared, or per use</h3>
<p>No numbers here, because they age badly and depend on your agreement. The shapes are what matter.</p>
<p>A Fabric capacity is reserved compute shared by every workload on it. Smoothing spreads background jobs over the next 24 hours, which makes steady loads pleasant. Overuse it, though, and Fabric first delays and then rejects interactive requests, and report users are the interactive requests (<a href="https://learn.microsoft.com/en-us/fabric/enterprise/throttling">throttling</a>). One oversized nightly job becomes a morning of slow reports. Separate capacities, or <a href="https://learn.microsoft.com/en-us/fabric/data-engineering/autoscale-billing-for-spark-overview">on-demand billing for Spark</a>, which takes Spark off the capacity, are the usual fixes.</p>
<p>Databricks bills compute as it runs, and idle compute can cost nothing. That suits spiky loads. The bite is ownership: without compute policies, auto-termination and tags, the bill is the sum of many small choices nobody reviews.</p>
<h3>5. Edge: streaming and machine learning</h3>
<p>Both platforms stream and both do machine learning. The question is how deep. Code-first ML with its own lifecycle, feature pipelines and model serving leans Databricks. Event data that business users explore and alert on leans toward Fabric's Real-Time Intelligence. If neither is on the roadmap, do not let a hypothetical one pick the platform.</p>
<h3>6. When the honest answer is both</h3>
<p>Two strong tilts in opposite directions, for example an engineering team with an existing Unity Catalog setup and a business that lives in Power BI, usually mean both. The pattern I trust: Databricks owns engineering and the tables; Fabric mirrors the gold catalog and owns semantic models and reports.</p>
<p>It works if three rules hold. Each table has one owner and one writer. Data flows one way per layer. And access is designed twice on purpose: Unity Catalog for the engineering side, Fabric permissions and OneLake security for consumers, reviewed together.</p>
<p>The reverse direction exists too. Azure Databricks can read Fabric Lakehouse and Warehouse tables through <a href="https://learn.microsoft.com/en-us/azure/databricks/query-federation/onelake">OneLake catalog federation</a>, read only, with several Fabric tenant settings to enable. Writing from Databricks into OneLake through an <a href="https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/external-locations-onelake">external location</a> is in Beta, and Learn warns that any non-viewer in that Fabric workspace can read, write and delete the data. I would not build a two-way flow on that yet.</p>
<p><img src="https://www.geekindata.com/assets/blogs/51-fabric-or-databricks-decision/both-one-way-flow.svg" alt="Both platforms together: Databricks owns ingestion, engineering and Unity Catalog; Fabric mirrors the gold catalog through shortcuts and serves Direct Lake semantic models and reports; OneLake federation lets Databricks read Fabric tables read only; access is designed separately on each side" /></p>
<p><em>Figure 4. One owner per table, one direction per layer, two permission models designed on purpose.</em></p>
<p><img src="https://www.geekindata.com/assets/blogs/51-fabric-or-databricks-decision/where-it-bites.svg" alt="Where each choice bites a year later: Fabric workspaces sprawl without Git discipline, shared capacity throttles reports, workspace roles bypass OneLake roles; Databricks setups stall without engineers, compute bills fragment without policies, Direct Lake needs a mirroring hop; both together duplicate governance and add sync latency" /></p>
<p><em>Figure 5. Pick the bites you can staff for.</em></p>
<h2>Common mistakes</h2>
<ul>
<li>Choosing by feature list. Both have the feature; the question is who will run it.</li>
<li>One capacity for engineering and reports. The nightly job slows the morning.</li>
<li>Databricks with no engineers. It ends up as edited notebooks and an unowned bill.</li>
<li>Assuming mirrored means governed. Unity Catalog grants stay behind; masked tables do not mirror.</li>
<li>Workspace Contributor as a convenience. It bypasses OneLake security roles.</li>
<li>"Both" without table ownership. Two writers, two truths.</li>
<li>Letting a future ML project decide today. Decide on what is on the roadmap.</li>
</ul>
<h2>Use as-is or adapt</h2>
<p><strong>Use as-is</strong></p>
<ul>
<li>The five questions, people first.</li>
<li>One owner and one writer per table when you run both.</li>
<li>Separate heavy engineering from report capacity, or take Spark off it.</li>
<li>Designing access on each side explicitly, never assuming it carries over.</li>
</ul>
<p><strong>Adapt before you use</strong></p>
<ul>
<li>The order of consumers and footprint, when what already exists is large.</li>
<li>How you split capacity, by workload or by environment.</li>
<li>Whether reports read a mirrored catalog or a copied gold layer, given sync latency and the row filter and column mask limits.</li>
<li>Whether you need the reverse flow at all.</li>
</ul>
<h2>Try it yourself</h2>
<p>The Fabric and Databricks demo objects for this post are in my public tech-demos repo, with a bundle and a compute policy you can deploy to rebuild the Databricks side.</p>
<p>Pick the platform your team can run, then design the seams on purpose.</p>
]]></content:encoded></item></channel></rss>