Ran into the same issue in three separate Fabric engagements over the last few months, so writing it up in case it saves someone time.

All newly created Fabric workspaces default to the writeHeavy Spark resource profile. This is correct for Bronze and ingestion workloads.

The problem: writeHeavy disables V-Order by default, and a lot of teams never change the profile on their Gold workspace.

What that costs you, per Microsoft Learn's cross-workload optimization guide:

• Direct Lake cold-cache queries: 40 to 60 percent slower without V-Order.
• SQL analytics endpoint and Warehouse: roughly 10 percent slower reads.
• Spark: no read impact either way.

The fix is one line of config per environment. Set spark.fabric.resourceProfile to readHeavyForPBI on Gold workspaces, readHeavyForSpark on Silver if it's read-heavy, leave writeHeavy on Bronze.

A few other things that surprised me while digging into this:

• Optimize Write, Auto Compact, and Low Shuffle Merge are all enabled by default in Fabric's Spark runtime. A lot of "optimization advice" on the internet is telling you to re-enable things that are already on.
• Liquid Clustering is the recommended approach for new tables, but ALTER TABLE ... CLUSTER BY on an existing unpartitioned table requires Delta Lake 3.3. Fabric Runtime 1.3 is on Delta 3.2. So you can create new clustered tables today, but retrofitting existing tables requires a migration.
• Runtime 2.0 (Spark 4.0, Delta 4.0) is Experimental Public Preview. Delta 4.0 features like type widening and variant type only work in Spark notebooks. If the table is read by Direct Lake, SQL endpoint, or Warehouse, those features break interoperability. Microsoft's own guidance is to stay on Runtime 1.3 for production.

I wrote up the full decision framework (five levers, per-layer profile mapping, Runtime 2.0 caveats) here if it's useful.

Curious what others have seen. Has anyone actually measured the V-Order difference on a real Direct Lake model, or is the 40 to 60 percent number holding up in practice?

Is there any reliable native way or 3rd party tool that can help me compare notebooks from one Workspace to another like checking if all notebooks available in one Workspace is valuable in another Workspace or not and if available their code block matches or not.

We are not using deployment pipeline so it's not option. I'm thinking about any python library or any tool

I have a case where we want to use Fabric without Power BI. We have a trial started, but when my Fabric developers want to create a workspace on the trial capacity, they are prompted to start a Power BI trial. They already have Fabric Free-licenses.

Is a Power BI license required to work in Fabric with notebooks, lakehouses etc (no PBI models)?

I have a single semantic model in my F2 capacity that seems to be consuming quite a bit of CU resources and throttling. I'm in the process of stripping it down to improve performance, but wondering if there is a set of strategies to systematically vet a semantic model and set it up for Direct Lake success? I currently process everything in notebooks and store in lakehouses with a final stored procedure to write to a Gold Layer Warehouse.

https://preview.redd.it/met0jqslclwg1.png?width=975&format=png&auto=webp&s=132db410bb9ca0eddd7a8e01efc61b62cf1abdb0

https://preview.redd.it/jgukranlclwg1.png?width=975&format=png&auto=webp&s=9cca3d04ef1462075293b81f7b2c3749288df9ff

https://preview.redd.it/7fm35anlclwg1.png?width=960&format=png&auto=webp&s=61fdc4f011c7099b0d1748a61ebd31e5aa9fb85d

How much CU overhead is there when running Notebook with Workspace Identity.

We have lots of available capacity. We have different users and developers starting new sessions and running pipelines and complex scripts. We stop all that.

I can run a test Pipeline that calls a Notebook.

Once I use a connection object with Workspace Identity auth for the notebook....and set up the permissions, we get:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook. Error: {"code":"BadRequest","subCode":0,"message":"Encountered internal error while calling TokenProvider to get obo token. The return code is BadRequest, and no error details was provided."

I do not believe this is a capacity issue. If it is, how much additional overhead would running the Notebook through a connection use? I can't image this would increase it. Does it? By how much?

Otherwise -- there is something broken in connections + Workspace Identity.

Is there a standard approach to get the security context (a guid that can be used to identify the account, user / account name) of the notebook execution?

I want to write that security context somewhere, so I want to know the standard way to retrieve it in consideration that the expected security context is different depending on how it’s run. I want to capture it though.

We're currently running the out of the box Fabric Capacity Metrics app on a Pro workspace (shared capacity), which limits us to 8 refreshes per day. Since we already have F128 capacities available, can we assign this workspace to our F128 capacity to unlock higher refresh limits, or are there other alternatives to increase the refresh frequency?

The fabric-cicd library just hit v1.0.0 — a huge milestone for the project! This release brings some significant changes, so if you're already using the library, please read through the breaking changes before upgrading. Let's break it all down.

Breaking Changes — Read Before Upgrading!

This release introduces some important breaking changes that require action on your part:

• Explicit token credential is now required. The default credential fallback has been removed. You must now explicitly pass a token credential — no more implicit authentication as a safety net.
• Implicit authentication in Fabric Notebook has been removed. Explicit token credential is also required when running fabric-cicd from the Fabric Notebook context.
• FabricWorkspace and deploy_with_config now require keyword-only arguments. Update your function calls to use explicit keyword arguments.
• Identity logging has been removed along with the disable_print_identity feature flag. Optionally remove the no longer existing flag.

These changes make authentication more intentional and secure. Please review the updated documentation for more information on supported authentication here: Getting Started - fabric-cicd.

New Item Support

• Ontology item type is now supported. New to the Fabric ecosystem, you can now deploy Ontology items as part of your CI/CD pipelines.

New Functionality

• Better deployment transparency: deploy_with_config now gives clearer feedback on success and failure scenarios, making it much easier to debug pipelines.
• Unpublish operations now collect API responses: API response collection has been extended to unpublish operations, giving you better visibility across the full deployment lifecycle.
• New get_changed_items() utility function: Detect Fabric items changed via git diff and use them for selective deployment — a great way to speed up your pipelines by only deploying what actually changed. Special thanks to u/vipulb91 for this awesome contribution!

Bug Fixes

Quite a few important fixes landed in this release:

• Prevent unintended GUID replacements in Variable Library item files during publish.
• Fix YAML content check to properly reject non-YAML files (like Notebooks) during key_value_replace parameterization, preventing file corruption.
• Fix Notebook deployment failures caused by non-deterministic ordering of definition files in the API payload (Convert py to ipynb failed with unexpected error).
• Parameter file is now ignored when not explicitly defined in the config file — no more surprise behavior.
• New enable_hard_delete feature flag to bypass the workspace recycle bin during unpublish operations.
• Addition of a timeout for long-running operation polling to prevent pipelines from hanging indefinitely.

Get Started

Check out the full release notes here: https://github.com/microsoft/fabric-cicd/releases/tag/v1.0.0

To upgrade:

pip install fabric-cicd==1.0.0

Relevant Links

• fabric-cicd GitHub
• fabric-cicd Docs
• Raise a GitHub Issue

Through deployment pipeline, promoting the Dev semantic model to UAT workspace but after deployment Data source setting is not swapping Uat warehouse (copied below screenshot). In the pipeline deployment Data Source rules also disabled. And i tried to parameritized the experession.tmdl with below code and it's not working. Finally ChatGPT said fabric won't support change expression.tmdl and finally said to use the deployment pipeline and automatically M code will swap from dev to uat warehouse connection, but it's not happening. How to promote the semantic model?

https://preview.redd.it/lzi70zsxclwg1.png?width=2322&format=png&auto=webp&s=256aa2d0c153674095f5dfa031bb0a3b8808045c

expression Environment = "DEV" meta [IsParameterQuery=true, IsParameterQueryRequired=true, Type=type text, List={"DEV","UAT","PROD"}, DefaultValue="DEV"]

lineageTag: ebbd659f-278f-45d5-a23d-53537661e1d1



annotation PBI\_ResultType = Text

expression 'DirectLake - fab_core_gld_dwh' =

	let

		Env = Environment,



		WorkspaceId = (

if Env = "DEV" then "yyyyyyy-16e7-yyyy-a5fd-yyyyyyy"

else if Env = "UAT" then "zzzzz-e834-4029-a3e4-cccccccc"

else if Env = "PROD" then "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"

else error "Invalid Environment"

		),



		LakehouseId = (

if Env = "DEV" then "xxxxxxxx-a9dc-44f1-b01b-xxxxxx"

else if Env = "UAT" then "yyyyyy-934f-tttttt-9b5b-zzzzzz"

else if Env = "PROD" then "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"

else error "Invalid Environment"

		),



		OneLakeURL =

"https://onelake.dfs.fabric.microsoft.com/"

& WorkspaceId & "/"

& LakehouseId,

		Source = AzureStorage.DataLake(

OneLakeURL,

[HierarchicalNavigation = true]

		)

	in

		Source

annotation PBI\_IncludeFutureArtifacts = False

Hello Fabric community,

what do you recommend for best practice? I have my work items like notebooks, pipelines etc. in one workspace for each stage (dev, test, prod), so a total of 3 workspaces.

Additionally, I don't want my lakehouses to be in the same workspace as the other items, so i stored my lakehouses and its data in another workspace. So my question is: Do you recommend one workspace for all lakehouses (Bronze, Silver, Gold) and every stage (dev, test, prod) is aligned to these central lakehouse workspace or is it better to also store the data seperately in dev, test, prod?

I’m currently working as a data/analytics engineer at a small/mid sized manufacturing company where I’ve more or less been tasked with building out our data platform from scratch. Also I’m the first data hire so I literally have no one to turn to in the company. This /r has been my guidance for everything lol. I’m also someone who does not have a lot of experience in DE. So I’m learning everything and then implementing it.

We’re still pretty early in our data maturity, with a lot of siloed systems (CRM, ERP, Sharepoint lists, etc.), so the goal has been to create something scalable but not overly complex.

Right now, I’ve set things up in Microsoft Fabric using a medallion-style architecture:

- Bronze (central lakehouse): raw data ingested from various APIs with minimal transformation

- Silver (central processing layer): cleaned and standardized using a config-driven pipeline

- Gold (department-level warehouses): business-ready tables in separate workspaces for different teams (Sales, Ops, etc.)

On top of that, I’m using workspace isolation so each department has its own workspace for reporting and access control, while keeping bronze and silver in a central workspace.

A lot of this is still evolving (e.g., handling schema changes, thinking about incremental loads vs overwrites, optimizing compute usage in Fabric, etc.), and I’m trying to strike a balance between doing things “right” and not over-engineering too early.

Curious to hear from others who’ve built something similar:

- Does this architecture make sense at this stage?

- Anything you’d strongly recommend changing early before it becomes painful later?

- How would you approach scaling this (especially around governance, costs, and team autonomy)?

Appreciate any thoughts or critiques.

Hi!
I am trying to set up a new CopyJob with audit columns. But I cannot find the option anywhere. Any idea where this option is hidden?

Instructions:
https://learn.microsoft.com/en-us/fabric/data-factory/audit-columns-copy-job

Blog post:

Audit columns in Copy job in Fabric Data Factory—every row is traceable for data lineage and compliance | Microsoft Fabric Blog | Microsoft Fabric

Screenshots along the process:

https://preview.redd.it/5zftub7sziwg1.png?width=1533&format=png&auto=webp&s=5bc330910b1ef4f8bb8bda2a878c1343c24638d8

https://preview.redd.it/t9mutd5dziwg1.png?width=1530&format=png&auto=webp&s=bc453cef6d1725eb58e581336c7c5f88c45a8f18

https://preview.redd.it/qicqf02uziwg1.png?width=1536&format=png&auto=webp&s=3d0f44f9d6142fc72f15960b857d90b0585defc9

https://preview.redd.it/bq8syt4vziwg1.png?width=1534&format=png&auto=webp&s=e31ecef797f217b7fd71e9bcfcba0f958821ffe2

https://preview.redd.it/axrczvcxziwg1.png?width=1539&format=png&auto=webp&s=4b7c5a5cfb59039abda3440af208b399f86125a4

https://preview.redd.it/8kvvagd00jwg1.png?width=1530&format=png&auto=webp&s=5b6f74691e388a444e03c6fdd42f68e12dcbd298

Hi everyone

We have a 6-month contract opening for an Analyst (Data Management) in my team. This role is open only to candidates currently based in Canada.

Looking for someone with 2+ years experience in data analytics/engineering, data quality/governance, and tools like SQL, Python, Apache Spark and Microsoft Fabric/Power BI.

If you or someone you know might be interested, DM me, I can share more details.

I’ve been experimenting with combining AI tools (specifically Claude) with Power BI semantic models, and it’s starting to feel like a meaningful shift in how analytics might be consumed.

Traditionally, the workflow looks something like this:

• Build reports and dashboards
• Add visuals to answer anticipated questions
• Users click, filter, and explore

That model works well, but it’s always limited by what was pre-built into the report.

What’s interesting now is the ability to connect an LLM directly to a semantic model and let users ask questions in natural language. Instead of navigating visuals, they can just ask things like:

• Why did June outperform March?
• What are the main drivers behind this trend?
• How do two entities compare and what should we do about it?

The responses aren’t just filtered views of existing visuals either. The model can:

• Pull in dimensions not surfaced in the report
• Provide summaries across multiple metrics
• Offer explanations and even recommendations

It starts to feel less like “report consumption” and more like interacting with the data itself.

A few things that stood out to me while testing this:

• The semantic model becomes way more important than the report layer
• You can’t realistically pre-build all the insights users might want
• AI fills the gap between high-level dashboards and deep analysis
• There’s real potential for scaling analytics to less technical users

That said, there are still some open questions:

• How do you manage trust and validation of AI-generated insights?
• What does governance look like when users can query anything?
• Does this reduce the need for complex report design, or just shift it?
• How do organizations handle access and security at scale?

Curious how others are thinking about this.

Are you exploring AI + Power BI integrations yet?
Do you see this replacing parts of the dashboard experience, or just augmenting it?

For Powerbi Developers team, we want to give the only pubish permission to reporting workspace. For publish, required to grant contribution permission. with contribution permsiion they can create another items i.e data pipelines etc. We want to restirct the item creation, For ex: Only they can publish the reports and rest all other item should be restrict. is it possible ?

Feels good to have the certification for credibility.

Followed the learning path on Microsoft learn.

Practiced a lot of mock tests available online. (Certiace, skillcerpro etc as I wanted to crack it in first attempt)

Generated lot of mock tests using Claude.

This week my BI Developer colleague proudly showed me a new Power BI report that he'd vibe-coded. Here's how it works:

1. Write a SQL query that selects the data needed for the report, concatinates it into one massive row, then format that row as a JavaScript array.
2. Write your custom report as a html web-page, complete with styles and JS functions.
3. Put the whole web page code file into one large string. Put the JS array containing your data from step 1 into your code string so that you now have a JS variable containing all of your raw data hardcoded into your html.
4. You now have a large string of html + JS that contains your custom report complete with data! Sadly the string exceeds the length of VARCHAR(MAX), so you'll need to chop it up, and insert each chunk into a table. Now all you need to do is set the table as a data source in PBI, re-join the rows into one long string, and voilà! A custome Power BI visual in 4 simple steps!

I'm fairly new to the data engineering role (transitioned from software dev) but this is insane right? My colleage has very strong SQL skills but isn't really a programmer, so I'm guessing this is a case of 'when all you have is a hammer, everything looks like a nail'.

I don't even know how to begin trying to explain the problems with this approach to my colleague, or what to suggest as an alternative (maybe just make a custom visual using the dev tools provided by PBI?). I don't want to come off sounding condescending but I have to say something before this becomes our standard way of creating custom reports.