diff --git a/content/en/docs/marketplace/platform-supported-content/modules/data-importer-extension.md b/content/en/docs/marketplace/platform-supported-content/modules/data-importer-extension.md index 280c292792c..50d609b5647 100644 --- a/content/en/docs/marketplace/platform-supported-content/modules/data-importer-extension.md +++ b/content/en/docs/marketplace/platform-supported-content/modules/data-importer-extension.md @@ -1,11 +1,15 @@ --- -title: "Data Importer" +title: "Data Importer Extension" url: /appstore/modules/data-importer/ description: "Overview of the Data Importer in Studio Pro" aliases: - /appstore/modules/data-importer-extension/ --- +{{% alert color="warning" %}} +For Studio Pro version 11.15 and above, see [Data Importer](/refguide/data-importer/). +{{% /alert %}} + ## Introduction {{% alert color="info" %}} diff --git a/content/en/docs/refguide/modeling/integration/data-importer.md b/content/en/docs/refguide/modeling/integration/data-importer.md new file mode 100644 index 00000000000..dd93fe52f45 --- /dev/null +++ b/content/en/docs/refguide/modeling/integration/data-importer.md @@ -0,0 +1,99 @@ +--- +title: "Data Importer" +url: /refguide/data-importer/ +weight: 40 +description: "Describes how to use Data Importer in Studio Pro to import data from Excel and CSV files." +#If moving or renaming this doc file, implement a temporary redirect and let the respective team know they should update the URL in the product. See Mapping to Products for more details. +--- + +{{% alert color="warning" %}}For Studio Pro version 11.14 and below, see the [Data Importer Extension](/appstore/modules/data-importer/).{{% /alert %}} + +## Introduction + +Data Importer lets you define how data from Excel and CSV files is interpreted in your Mendix app. You create a Data Importer document based on an input file. + +The document can be used in two ways: + +* With the **Import data from file** activity, to import the file into a generated non-persistable entity. +* As a source for an import mapping, which gives you more control over how imported data is mapped to Mendix objects. + +## Creating a Data Importer Document + +To create a Data Importer document, follow the steps below: + +1. Right-click the module where you want to add the Data Importer document, then click **Add other** > **Data Importer**. +2. Enter a name for the document, then click **OK**. + +The new Data Importer document opens. + +{{% alert color="warning" %}}Run the app before you configure the file settings or preview data.{{% /alert %}} + +## Previewing Data + +After creating the Data Importer document, click **Upload File** in **Select file from local** to upload an Excel file (*.xls* or *.xlsx*) or CSV file (*.csv*). You can upload a file up to 10 MB size. + +An Excel workbook can have one or multiple sheets. Choose which sheet to import data from and configure the Excel file settings below: + +* **Sheet Name** – the name of the worksheet to import. If the workbook has multiple worksheets, their names appear in the drop-down list. +* **Header Row No.** – row number of the file header; the default is 1. +* **Read Data From** – the row where data reading starts; the default is 2. + +CSV import supports multiple combinations of delimiter, quote, and escape characters. It also supports files without a header row. Configure the following settings: + +* **Delimiter (Separator)** – Supported delimiters are comma, semicolon, pipe, and tab. The default is comma. +* **Quote Characters** – Supported quote characters are single quotes and double quotes. The default is double quotes. +* **Add Header Row** – Specify whether to add a header row or whether the CSV file already includes one. By default, the file already includes a header row. +* **Escape Character** – Supported escape characters are backslash, single quotes, and double quotes. The default is double quotes. + +Click **Preview Data** to view the data from the selected file. + +Data Importer creates the data structure based on the first ten rows of the source file and displays it in the **Structure elements** section. If the file settings do not provide valid data, an error is displayed. To modify **Custom Name** or **Primitive Type**, click the edit icon ({{% icon name="pencil" %}}) in the bottom-right corner of the structure elements table. + +{{% alert color="warning" %}} +Column names that do not adhere to Mendix naming conventions are autocorrected. For Number cell types, the target Mendix type is mapped to **Decimal** to support both integers and decimals. +{{% /alert %}} + +You can now use the data importer document in the import mapping. For more information, see the [Using in an Import Mapping](#using-import-mapping) section below. + +## Editing an Entity + +Optionally, if you are not using an import mapping, you can create a mapping flow first by adjusting the entity structure in the **Entity Preview** section before creating the entity. + +Click the edit icon ({{% icon name="pencil" %}}) in the bottom-right corner of **Entity Preview**. In the dialog box: + +* Change the entity **Name**. +* Rename attributes: **Original Name** shows the column name from the input file, and **Attribute Name** is the new name you want to assign to that column. +* Change the data type of an attribute by selecting a value from the drop-down list. + +In the **Entity Preview**, select which columns to import by selecting or clearing the checkbox next to each attribute. + +Click **OK** to save your changes, or click **Cancel** to discard them. + +{{% alert color="warning" %}} +**Enum** is not supported as a target data type. Runtime exceptions can occur if the input data cannot be converted to the target data type, for example because of invalid data, data truncation, or casting issues. +{{% /alert %}} + +## Creating an Entity + +After reviewing the entity structure in **Entity Preview**, click **Create Entity**. This creates the entity in your domain model and displays a confirmation message. The Data Importer document is then ready to use in [Import Data from File](/refguide/import-data-from-file/) to import data. For more details, see the [Using in the Import Data from File Activity](#using-in-the-activity) section below. + +To change the source file at any point, click **Remove File**, upload a new file, and reconfigure the document. Note that removing the file clears all structure elements and configured mappings. + +## Using a Data Importer Document + +You can use a Data Importer document in two ways: + +* For simple use cases, directly in the **Import data from file** activity to import data into non-persistable entities (NPEs). +* As the schema source for an import mapping, when you need more control over how data is mapped to Mendix objects. + +### Using in the Import Data from File Activity {#using-in-the-activity} + +After creating the entity, you can use the Data Importer document in the [Import Data from File](/refguide/import-data-from-file/) activity to import data into a list of NPEs. + +You can extend this further. For example, you can convert the list of NPEs into persistable entities by providing a message definition, or use a loop to create and commit entities to your database individually. + +### Using in an Import Mapping {#using-import-mapping} + +After the document is created and its **Structure elements** are populated, you can use the Data Importer document in an import mapping by selecting **Excel/CSV Structure** as the **Schema source**. The structure elements defined in the Data Importer document become the schema that you map to your Mendix entities and attributes. + +This approach gives you more control than the `Import data from file` activity. You can map imported data to existing persistable entities, find existing objects by key instead of always creating new ones, set associations between mapped objects, and apply conversion microflows to transform attribute values during import. For more information, see [Import Mappings](/refguide/import-mappings/). diff --git a/content/en/docs/refguide/modeling/integration/mapping-documents/select--elements.md b/content/en/docs/refguide/modeling/integration/mapping-documents/select--elements.md index e2e4b82ef3c..11b772a0b22 100644 --- a/content/en/docs/refguide/modeling/integration/mapping-documents/select--elements.md +++ b/content/en/docs/refguide/modeling/integration/mapping-documents/select--elements.md @@ -11,7 +11,7 @@ aliases: For both [import](/refguide/import-mappings/) and [export mappings](/refguide/export-mappings/), you need to specify the elements structure you want to map. You do this in the **Select schema elements** window. An example of this screen is shown below: -{{< figure src="/attachments/refguide/modeling/integration/mapping-documents/select--elements/schema-elements-window.png" class="no-border" >}} +{{< figure src="/attachments/refguide/modeling/integration/mapping-documents/select--elements/select-schema-elements.png" >}} Perform the following steps in the **Select schema elements** window: diff --git a/content/en/docs/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer.md b/content/en/docs/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer.md deleted file mode 100644 index 49693b837d2..00000000000 --- a/content/en/docs/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer.md +++ /dev/null @@ -1,221 +0,0 @@ ---- -title: "Use the Data Importer" -url: /refguide/use-the-data-importer/ -weight: 21 -description: "Overview of the Data Importer in Studio Pro" -aliases: - - /howto/integration/use-the-data-importer/ -#If moving or renaming this doc file, implement a temporary redirect and let the respective team know they should update the URL in the product. See Mapping to Products for more details. ---- - -## Introduction - -Data is constantly exchanged between various systems inside and outside an organization. The most commonly used file formats for data exchange are Microsoft Excel and comma-separated value (CSV). These files contain data in a tabular grid of rows, columns, and delimiter-separated values. - -This how-to teaches you to do the following: - -* Create a Data Importer document using a sample representative file (Excel and CSV) -* Create a (non-persistable) entity in your domain model -* Import data using the custom **Import data from file** activity - -## Prerequisites - -Download the [Data Importer extension](https://marketplace.mendix.com/link/component/219833) from the Marketplace and [add it into your app](/appstore/use-content/#install). This module also requires a file document (for more information, see [File Manager](/refguide/file-manager/)) - -## Data Importer Document - -The Data Importer extension allows you to import data from Excel and CSV files directly into your app. Create a Data Importer document to define which columns to import and a non-persistable entity (NPE) to hold the imported data, along with source-to-target mapping. During the Data Importer document creation, you can preview the data and choose which columns you want to import and edit the name of resulting entity. - -The Data Importer document can be used along with the [Import data from file](/refguide/import-data-from-file/) custom activity. Use this activity in a microflow to import data from an Excel or CSV file. - -### Creating a Data Importer Document - -Right-click the module you want to add the Data Importer document to and click **Add other** > **Data Importer**. - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/data-importer-menu.png" class="no-border" width="600" >}} - -Name the document, then click **OK**, and the new Data Importer document opens. - -### Previewing Excel Data {#preview-excel-data} - -Click **Select a local file** to import an Excel file (*.xls* or *.xslx*). - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/select-file-for-preview.png" class="no-border" width="600" >}} - -Select or drop the file in the **Select Source File** field. An Excel workbook can have single or multiple sheets; you can choose which sheet to import data from and specify the header row and starting data row. - -* **Sheet Name** – name of the worksheet from where data needs to be imported; if the Excel has multiple worksheets, their names will appear in the dropdown -* **Header Row No.** – row number of the file header; the default is 1 -* **Read Data From Row No.** – starting line for reading data; the default is 2 - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/select-sheet-and-header-data-row.png" class="no-border" width="600" >}} - -Click **Preview Source Data & Entity** to view the data from the file. The first 10 data rows from the source file are shown in the data preview section. If there are less than 10 data rows in the sample file, only the available rows are shown. The column names correspond to the attribute name within the entity, and the sheet name is used to define the entity. - -All the columns are automatically selected (checked) for import. You can uncheck the columns you do not want to use. At the bottom of the table, you see the target data type of the attribute, which is based on the cell-type defined in the Excel file's first data row. If any data types are incorrect, check the cell-type of the first data row and adjust the definition accordingly. - -{{% alert color="warning" %}} Column names that do not adhere to Mendix naming conventions will be autocorrected. For **Number** cell-types, the target Mendix type is mapped to **Decimal** to accommodate to integers and decimals. {{% /alert %}} - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/preview-data-and-entity.png" class="no-border" width="600" >}} - -### Previewing CSV Data {#preview-csv-data} - -Select or drop the CSV file in the **Select Source File** window. CSV import supports multiple combinations of separator/delimiter, quote, and escape characters. It also supports importing files where the header row is absent. - -Specify the values for all four configurations (Delimiter, Quote Character, Escape Character, and Add Header Row): - -* **Delimiter (Separator)** – current supported delimiters are comma, semicolon, pipe, and tab; the default is comma -* **Quote Characters** – current supported quote characters are single and double quotes; the default is double quotes -* **Escape Characters** – current supported escape characters are backslash, single, and double quotes; the default is double quotes -* **Add Header Row** – specify if you want to add a header row or if the header row is already part of the CSV file; the default is the header row already included in file - -Click **Preview Source Data & Entity** to view the data from the file. The first ten rows from the source file are shown in the data preview section. The file name is used to define the entity (NPE), but this can be edited. The column names correspond to the attribute name within the entity. - -All the columns are selected (checked) by default. You can uncheck the columns you do not want to import. At the bottom of the table, you can see the target data type of the attribute, which defaults to **String**. - -{{% alert color="warning" %}} Column names that do not adhere to Mendix naming conventions will be autocorrected. {{% /alert %}} - -For example, for the following source data (CSV), the separator is specified as Comma and Quote, and the Escape Character is Double Quote and Header. This is already part of the input file. - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/source-csv-data.png" class="no-border" width="600" >}} - -The data preview and resulting entity are seen below: - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/preview-csv-data-and-entity.png" class="no-border" width="600" >}} - -### Editing an Entity {#edit-entity} - -You can edit the entity in the **Entity Preview** section. The Data Importer supports various ways to: - -* Edit the name of resultant entity -* Edit the name of the attribute (or attributes) of the entity -* Edit the data type of a given attribute - -Click **Edit** at top-right corner of **Entity Preview**. This will render a pop-up window where you can change the name of the entity. You can also change the name of the attribute; *Original Name* is the name of the column from input file and *Attribute Name* will be the new name that you want to assign to this column. You can also change the data type of this attribute by selecting a relevant value from the drop-down as shown below. - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/edit-csv-entity.png" class="no-border" width="600" >}} - -Once you are satisfied with the changes, click **OK** to save or **Cancel** to discard your changes. - -{{% alert color="info" %}} -The **Edit Entity** feature is useful for CSV import, as all the columns of a CSV file are marked as String by default, so you can change the data type if necessary. The following table shows the source-to-target data conversion matrix: - -Input CSV File - -| Source Type | Target- String | Target- Int | Target- Long | Target- Decimal | Target- Boolean | Target- DateTime | -| :-------- | :------- | :-------- | :------- | :-------- | :------- | :-------- | -| String | Yes | Partial | Partial | Partial | Partial | No | - -Input Excel File - -| Source Type | Target- String | Target- Int | Target- Long | Target- Decimal | Target- Boolean | Target- DateTime | -| :-------- | :------- | :-------- | :------- | :-------- | :------- | :-------- | -| String | Yes | Partial | Partial | Partial | Partial | No | -| Boolean | Yes | No | No | No | Yes | No | -| Decimal | Yes | Partial | Partial | Yes | No | No | -| DateTime | Yes | No | No | No | No | Yes | - -**Partial** - If source data is valid and within range, it will be converted into the target data type. - -{{% /alert %}} - -{{% alert color="warning" %}} - -* **Enum** is not supported as a target data type -* Runtime exceptions can occur if the input data cannot be converted into desired the target data type for various reasons (for example, invalid data, data truncation, casting etc.) -{{% /alert %}} - -### Creating an Entity {#create-entity} - -When you are done editing the entity, click **Create Entity** > **OK**. This will create the entity in your domain model. You will also see a confirmation message that an entity has been created in the domain model and is ready to use. - -When the entity is created, you can view the mapping of the source columns to the target entity attributes. - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/source-to-target-mapping.png" class="no-border" width="600" >}} - -The Data Importer document creation is complete and can be used to import data in a microflow. - -## Building your App {#build-data-importer-app} - -The newly-created Data Importer document allows you to periodically import data from an Excel or CSV file that is generated by another app or system. - -### Custom Activity {#Import-data-from-file} - -The **Import data from file** activity is found under **Integration activities** in the **Toolbox**. Double-click to view its properties: - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/custom-activity-params.png" class="no-border" width="600" >}} - -The **Input** section includes: - -* **File** – name of the file from which you want to import data -* **Data Importer document** – the Data Importer document created at the end of the design time flow - -The **Output** section includes: - -* **Return Type** – set to the list of NPEs defined in the Data Importer document -* **Variable name** – auto-populated to the **EntityName** list - -### Build the Pages - -The **Import data from file** custom activity needs an input file to import data from. The example below builds a page where a `System.FileDocument` is uploaded and fed to the custom activity. - -1. Open the home page and add a button and name it *Upload Customer Data*. -2. Double-click the button and in the **Events** field under the **On click** drop-down, select **Create object** to create a `System.FileDocument` entity. -3. Pass the control to a new page (**UploadCustomerData**) where the file is uploaded. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/home-page-button.png" class="no-border" width="600" >}} - -4. On the **UploadCustomerData** page, include a data view for the *FileDocument* and include a 'File Manager' to assist with a file upload. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/data-view-file-manager.png" class="no-border" width="600" >}} - -5. Open the **Toolbox** and add a **Call microflow button**. - -6. Click **New** and name the microflow *Import Customer Data*. You also see **FileDocument** in the parameters section; make sure this box is checked to include it as a parameter and click **OK**. - -{{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/add-parameter.png" class="no-border" width="600" >}} - -### Configuring the Import data from file Activity in a Microflow - -{{% alert color="info" %}} -The steps below are shown using an Excel input file with its corresponding Data Importer document. You can substitute an Excel document with a CSV document to import data from CSV input files. -{{% /alert %}} - -1. In the created microflow, drag the **Import data from file** activity into it. You can find this activity in the **Toolbox** under **Integration activities**. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/integration-activity.png" class="no-border" width="600" >}} - -2. When the **Import data from file** activity is added into microflow, you see three errors in the console: - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/custom-activity.png" class="no-border" width="600" >}} - - To address these errors, double-click the activity and in the **File** field, choose the input file that is passed from the file upload page to this microflow as a parameter. - -3. In the **Data Importer document** field, click **Select** and choose the Data Importer document you want to use. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/choose-data-importer-template.png" class="no-border" width="600" >}} - - After selecting the Data Importer document, the **Return type** and **Variable name** auto-populates. You can change the name of the output variable if you wish. - -4. Click **OK**. The custom activity is configured and all the errors will resolve. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/configured-custom-activity.png" class="no-border" width="600" >}} - -5. Add an **Aggregate list** activity and configure it to count the size of the 'CustomerList', which is returned from the previous activity. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/aggregate-list.png" class="no-border" width="600" >}} - -6. Configure a **Show message** activity. You can use a template message and a parameter, such as in the example below. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/show-message-with-list-size.png" class="no-border" width="600" >}} - -7. Set '$CustomerList' as the return value from the **Import data from file** activity to be used later. Your completed microflow should look like the image below. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/example-microflow.png" class="no-border" width="600" >}} - -8. Deploy your app locally. Browse and upload an input file, which is similar to the file that was used as a template while creating Data Importer document. -9. Check that you see a message that states **Imported xx rows from input file into a list of NPEs**. - - {{< figure src="/attachments/refguide/modeling/integration/use-platform-supported-content/use-the-data-importer/local-app-run.png" class="no-border" width="600" >}} - -You have successfully configured and used the Data Importer extension. You can extend this as per your requirements. For example, you can convert the list of NPEs into persistable entities by providing a message definition, or use each loop construct and individually create and commit entities into your database. diff --git a/static/attachments/refguide/modeling/integration/mapping-documents/select--elements/schema-elements-window.png b/static/attachments/refguide/modeling/integration/mapping-documents/select--elements/schema-elements-window.png deleted file mode 100644 index ec09ef8766e..00000000000 Binary files a/static/attachments/refguide/modeling/integration/mapping-documents/select--elements/schema-elements-window.png and /dev/null differ diff --git a/static/attachments/refguide/modeling/integration/mapping-documents/select--elements/select-schema-elements.png b/static/attachments/refguide/modeling/integration/mapping-documents/select--elements/select-schema-elements.png new file mode 100644 index 00000000000..3d064e8bc0d Binary files /dev/null and b/static/attachments/refguide/modeling/integration/mapping-documents/select--elements/select-schema-elements.png differ