> ## Documentation Index
> Fetch the complete documentation index at: https://domoinc-openapi-sync-dataflows.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Magic ETL on BigQuery (Beta)

### Intro

Magic ETL on BigQuery allows you to execute your BigQuery DataSets using Domo data transforms and write results back to BigQuery without moving data or creating complex configurations.

***

### Prerequisites

To use this feature, you need a Domo Cloud Amplifier integration configured for both "read" and "write" capability with your BigQuery account.

Learn more about integrating [BigQuery using Cloud Amplifier.](https://domo-support.domo.com/s/article/000005237?language=en_US)

### Required Grant

To use Magic ETL on BigQuery, users must have the following grant enabled for their role:

* **Edit Adrenaline DataFlow —** Allows users to create, edit, and run Adrenaline DataFlows to which they have access.

### Access Magic ETL on BigQuery

You can use this feature in the Magic ETL interface after completing your BigQuery-Cloud Amplifier transform integration.

Access Magic ETL from the details page of any DataSet by selecting **Open With** > **Magic ETL**.

###

<Frame>
  <img alt="open with magic.jpg" src="https://mintcdn.com/domoinc-openapi-sync-dataflows/g5Ek3SRXv8ussuBG/images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000000w57N.jpg?fit=max&auto=format&n=g5Ek3SRXv8ussuBG&q=85&s=a2abc6fa61a51e657be8e94936656bbc" style={{width: 875, height: 306}} width="3168" height="1108" data-path="images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000000w57N.jpg" />
</Frame>

### Use Magic ETL on BigQuery

Magic ETL on BigQuery allows you to use the Magic ETL interface to *define* a data transformation, but the transformation *executes* on BigQuery.

When you configure a BigQuery-Cloud Amplifier integration to allow transform operations, a dropdown displays in the Magic ETL editor, where you can choose where the ETL executes. The dropdown provides options for each of the available BigQuery DataSets.

Follow these steps to define and execute a data transformation:

1. Find your BigQuery integration using the connection name. You can find the connection by navigating to **Data** > **Warehouse.**

   <Frame>
     <img alt="Screenshot 2024-09-23 at 7.58.27 AM.png" src="https://mintcdn.com/domoinc-openapi-sync-dataflows/g5Ek3SRXv8ussuBG/images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000002qVGH.jpg?fit=max&auto=format&n=g5Ek3SRXv8ussuBG&q=85&s=d7a74f17e9aa7461009c709b6dca8647" style={{width: 901, height: 461}} width="2246" height="1149" data-path="images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000002qVGH.jpg" />
   </Frame>
2. Select **Transform Data** > **Magic ETL.**

   <Frame>
     <img alt="Screenshot 2024-09-23 at 8.00.57 AM.png" src="https://mintcdn.com/domoinc-openapi-sync-dataflows/g5Ek3SRXv8ussuBG/images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000002qVhh.jpg?fit=max&auto=format&n=g5Ek3SRXv8ussuBG&q=85&s=afed83ffd5225cda65c43ff108ce8f4c" style={{width: 942, height: 477}} width="2233" height="1131" data-path="images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000002qVhh.jpg" />
   </Frame>
3. Use **Compute** to select your BigQuery integration. This action updates the Magic ETL background to indicate that you are now using Magic ETL on BigQuery.

   <Frame>
     <img alt="Screenshot 2024-09-23 at 8.12.09 AM.png" src="https://mintcdn.com/domoinc-openapi-sync-dataflows/g5Ek3SRXv8ussuBG/images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000002qXI5.jpg?fit=max&auto=format&n=g5Ek3SRXv8ussuBG&q=85&s=3a1dee492323f9a8ee9668f66c2bec31" style={{width: 976, height: 376}} width="2245" height="865" data-path="images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000002qXI5.jpg" />
   </Frame>
4. Select and drag an Input DataSet tile and the DataSet that is connected to the selected BigQuery integration.

   <Frame>
     <img alt="Screenshot 2024-09-23 at 8.28.29 AM.png" src="https://mintcdn.com/domoinc-openapi-sync-dataflows/g5Ek3SRXv8ussuBG/images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000002qZdF.jpg?fit=max&auto=format&n=g5Ek3SRXv8ussuBG&q=85&s=1c4f651489d1aa7a32c1dd59c2d8a492" style={{width: 1015, height: 502}} width="1994" height="986" data-path="images/kb/ka0Vq00000032LR-00N5w00000Ri7BU-0EMVq000002qZdF.jpg" />
   </Frame>
5. Finish creating your DataFlow. Every time your new DataFlow runs, the execution happens in BigQuery.

### FAQ

<AccordionGroup>
  <Accordion title="Can I set my Magic ETL DataFlow to use append when it executes on BigQuery?">
    No, currently, only 'replace' is supported.
  </Accordion>

  <Accordion title="Can I set my Magic ETL DataFlow to use upsert or partitioning when it executes on BigQuery?">
    No, upserts and partitions are not supported with Magic ETL on BigQuery.
  </Accordion>

  <Accordion title="Can my Magic ETL use existing partitions on my BigQuery table when running an ETL?">
    Yes, it executes like any other query.
  </Accordion>

  <Accordion title="Do I have to configure any special permissions in BigQuery to be able to use Magic ETL?">
    No, the necessary permissions will already exist when you set up write integration.
  </Accordion>

  <Accordion title="Can I limit which users in Domo can execute Magic ETL on BigQuery?">
    Yes, only Domo users with the "Edit Adrenaline DataFlow" grant can perform these executions.
  </Accordion>

  <Accordion title="Do I need to set up a default storage cloud?">
    No.
  </Accordion>

  <Accordion title="Can I take a Magic ETL DataFlow that was previously configured to execute on Magic ETL and swap it over to execute on my BigQuery cloud (assuming all inputs are on the BigQuery cloud)?">
    If the DataFlow was set up to upsert or partition, an error will occur. Otherwise, yes.
  </Accordion>

  <Accordion title="Is the Python tile supported?">
    No, the Python tile is not supported with this integration.
  </Accordion>

  <Accordion title="Is the R Scripting tile supported?">
    No, the R Scripting tile is not supported with this integration.
  </Accordion>

  <Accordion title="Am I charged Domo credits for a Magic ETL DataFlow that executes on BigQuery?">
    Yes, please refer to your sales agreement for credit consumption terms.
  </Accordion>
</AccordionGroup>