Synchronizing AWS for the first time
In this guide, we’ll walk you through an example of how to connect Raito Cloud to your AWS S3 (Glue) warehouse through the Raito CLI. We’ll
- make sure that Raito CLI is installed and available
- log into Raito Cloud and create a data source. Optionally we create an organization identity store
- configure Raito CLI to connect to Raito Cloud and synchronize with the previously-created data source
- run a first sync
For this guide, you will need access to Raito Cloud and you also need access to AWS. We assume that you’re able to create an AWS IAM user or AWS IAM role that can be used by the Raito CLI.
Raito CLI installation
To install the Raito CLI, simply run the following command in a terminal window:
$> brew install raito-io/tap/cli
Check that everything is correctly installed by running
$> raito --version
If you want more information about the installation process, or you need to troubleshoot an issue, you can find more information here.
Create an organization identity store (optional)
If you have an AWS organization set up and want to use permission sets to manage access, you have to sync the users and groups defined in IAM Identity Center. In that case, are required to create a new Identity Store.
In the left navigation pane, go to Identities > Identity Stores. You should see a button on the top-right named Add Identity Store. This will guide you through a short wizard to create a new identity store. The main things that you will need to configure are:
Identity store type. Select AWS Organization.Identity store name. Give your identity store a good descriptive name, separating it from other identity stores. For this example, we’ll choose ‘AWS Organization’.Identity store description. Accompany your identity store with a meaningful description.
Create an AWS account Data Source
To create a new Raito Data source, go to Data Sources > All data sources, in the left navigation pane. You should see a button on the top-right named Add data source. This will guide you through a short wizard to create a new data source. The main things that you will need to configure are:
Data source type. Select AWS.Data source name. Give your data source a good descriptive name, separating it from other data sources. For this example, we’ll choose ‘AWS Test account’.Data source description. Accompany your data source with a meaningful description.Connection method. Select whether you want to use the Raito hosted cloud version of the CLI or one managed by yourself, which is recommended. In this example we indeed select ‘CLI’.
If you created an organization identity store, you need to link the previously created IS with the newly created data source.
Navigate to the newly created data source. In the action menu (clicking the 3 dots on the top right), you should see the ability to Link to identity stores.
Add the AWS organization identity store you created in the section before and apply the changes.
AWS credentials
To connect to AWS, you need to provide the Raito CLI with the necessary credentials.
The AWS Account connector requires credentials to the account containing the S3 resources you want to manage access to.
Preferably you create a new profile in your ~/.aws/credentials file. More information can be found here
Ensure the policy defined on the AWS Account connector is attached to the role/user.
When using the optional AWS organization sync, credentials to the master account are required. Those credentials should be available by using another profile. The following policy should be attached to the user/role connecting to the master account of your organization:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ssoInstanceReadWrite",
"Effect": "Allow",
"Action": [
"sso:CreatePermissionSet",
"sso:DescribePermissionSet",
"sso:DescribePermissionSetProvisioningStatus",
"sso:GetInlinePolicyForPermissionSet",
"sso:GetPermissionsBoundaryForPermissionSet",
"sso:GetPermissionSet",
"sso:ListAccountsForProvisionedPermissionSet",
"sso:ListCustomerManagedPolicyReferencesInPermissionSet",
"sso:ListManagedPoliciesInPermissionSet",
"sso:ListPermissionSetProvisioningStatus",
"sso:ListPermissionSets",
"sso:ListPermissionSetsProvisionedToAccount",
"sso:DeleteInlinePolicyFromPermissionSet",
"sso:DeletePermissionSet",
"sso:ProvisionPermissionSet",
"sso:PutInlinePolicyToPermissionSet",
"sso:AttachCustomerManagedPolicyReferenceToPermissionSet",
"sso:AttachManagedPolicyToPermissionSet",
"sso:DeletePermissionsBoundaryFromPermissionSet",
"sso:DetachCustomerManagedPolicyReferenceFromPermissionSet",
"sso:DetachManagedPolicyFromPermissionSet",
"sso:PutPermissionsBoundaryToPermissionSet",
"sso:UpdatePermissionSet",
"sso:TagResource",
"sso:ListAccountAssignments",
"sso:ListAccountAssignmentsForPrincipal",
"sso:CreateAccountAssignment",
"sso:DeleteAccountAssignment"
],
"Resource": [
"arn:aws:sso:::instance/ssoins-${instanceId}"
]
},
{
"Sid": "ssoPermissionSetReadWrite",
"Effect": "Allow",
"Action": [
"sso:ListAccountAssignments",
"sso:ListAccountsForProvisionedPermissionSet",
"sso:ListManagedPoliciesInPermissionSet",
"sso:ListTagsForResource",
"sso:DescribePermissionSet",
"sso:GetInlinePolicyForPermissionSet",
"sso:GetPermissionsBoundaryForPermissionSet",
"sso:GetPermissionSet",
"sso:CreateAccountAssignment",
"sso:DeleteAccountAssignment",
"sso:DeletePermissionSet",
"sso:ProvisionPermissionSet",
"sso:AttachCustomerManagedPolicyReferenceToPermissionSet",
"sso:AttachManagedPolicyToPermissionSet",
"sso:DeletePermissionsBoundaryFromPermissionSet",
"sso:DeletePermissionsPolicy",
"sso:DetachCustomerManagedPolicyReferenceFromPermissionSet",
"sso:DetachManagedPolicyFromPermissionSet",
"sso:PutPermissionsBoundaryToPermissionSet",
"sso:PutPermissionsPolicy",
"sso:UpdatePermissionSet",
"sso:TagResource",
"sso:DeleteInlinePolicyFromPermissionSet",
"sso:ListCustomerManagedPolicyReferencesInPermissionSet",
"sso:PutInlinePolicyToPermissionSet",
"sso:ListPermissionSetProvisioningStatus",
"sso:ListPermissionSets",
"sso:ListPermissionSetsProvisionedToAccount",
"sso:DescribePermissionSetProvisioningStatus"
],
"Resource": [
"arn:aws:sso:::permissionSet/ssoins-${instanceId}/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceTag/creator": "RAITO"
}
}
},
{
"Sid": "ssoPermissionSetCreate",
"Effect": "Allow",
"Action": [
"sso:CreatePermissionSet"
],
"Resource": [
"arn:aws:sso:::permissionSet/ssoins-${instanceId}/*"
],
"Condition": {
"StringEquals": {
"aws:RequestTag/creator": "RAITO"
}
}
},
{
"Sid": "ssoPermissionSetTag",
"Effect": "Allow",
"Action": [
"sso:TagResource"
],
"Resource": [
"arn:aws:sso:::permissionSet/ssoins-${instanceId}/*"
],
"Condition": {
"StringEquals": {
"aws:ResourceTag/creator": "RAITO",
"aws:RequestTag/creator": "RAITO"
}
}
},
{
"Sid": "ssoListTags",
"Effect": "Allow",
"Action": [
"sso:ListTagsForResource"
],
"Resource": [
"arn:aws:sso:::permissionSet/ssoins-${instanceId}/*",
"arn:aws:sso:::instance/ssoins-${instanceId}"
]
},
{
"Sid": "accountAssignment",
"Effect": "Allow",
"Action": [
"sso:CreateAccountAssignment",
"sso:DeleteAccountAssignment",
"sso:ProvisionPermissionSet",
"sso:ListAccountAssignments"
],
"Resource": [
"arn:aws:sso:::account/*"
]
},
{
"Sid": "identitystoreRead",
"Effect": "Allow",
"Action": [
"identitystore:ListGroupMemberships",
"identitystore:ListGroupMembershipsForMember",
"identitystore:ListGroups",
"identitystore:ListUsers",
"identitystore:DescribeGroup",
"identitystore:DescribeGroupMembership",
"identitystore:DescribeUser",
"identitystore:GetGroupId",
"identitystore:GetGroupMembershipId",
"identitystore:GetUserId",
"identitystore:IsMemberInGroups"
],
"Resource": [
"*"
]
},
{
"Sid": "enrichRoles",
"Effect": "Allow",
"Action": [
"sso:ListInstances",
"sso:ListPermissionSets",
"sso:DescribePermissionSet",
"sso:ListAccountAssignments"
],
"Resource": [
"*"
]
},
{
"Sid": "identitycenter",
"Effect": "Allow",
"Action": [
"sso:ListInstances",
"sso:DescribeInstance"
],
"Resource": [
"arn:aws:sso:::instance/*"
]
},
{
"Sid": "identitystore",
"Effect": "Allow",
"Action": [
"identitystore:ListGroups",
"identitystore:ListUsers",
"identitystore:ListGroupMemberships",
"identitystore:ListGroupMembershipsForMember",
"identitystore:DescribeGroup",
"identitystore:DescribeGroupMembership",
"identitystore:DescribeUser",
"identitystore:GetGroupId",
"identitystore:GetGroupMembershipId",
"identitystore:GetUserId",
"identitystore:IsMemberInGroups"
],
"Resource": [
"*"
]
}
]
}
Raito CLI Configuration
To configure the Raito CLI to synchronize your AWS S3 warehouse, start by creating a file with the name raito.yml and edit it to look like this:
api-user: "{{RAITO_USER}}"
api-secret: "{{RAITO_API_KEY}}"
domain: "{{DOMAIN}}"
targets:
# Optional aws organization identity store sync if organization is enabled
- name: aws-organization
connector-name: raito-io/cli-plugin-aws-organization
identity-store-id: "<identity-store-id-of-organization-identity-store>"
aws-account-id: "<master account id>"
aws-profile: "<aws-profile connecting to the master account>"
aws-region: "<aws region of the identity center>"
# End optional config
- name: aws-account
connector-name: raito-io/cli-plugin-aws-account
data-source-id: "<data-source-id>"
identity-store-id: "<identity-store-id>"
aws-profile: "<aws-profile connecting to the account to sync>"
aws-account-id: "<account id>"
aws-regions: "<comma separated list of regions to sync>"
# Optional configuration if organization is enabled
aws-organization-profile: "<master account id>"
aws-organization-region: "<aws region of the identity center>"
aws-organization-identity-center-instance-arn: "<arn of the identity-center instance>"
aws-organization-identity-store: "<identity store id of the organization identity store>"
# End optional config
# Pointing to the S3 bucket where cloudtrail data is stored. Data usage information will be fetched from here
aws-s3-cloudtrail-bucket: "raito-cloudtrail"
# Optionally exclude buckets to be handled
aws-s3-exclude-buckets: "raito-cloudtrail,cdk-hnb659fds-assets-077954824694-eu-central-1"
# Enabled either the S3 or the glue option to fetch the data objects from one of these sources
aws-s3-enabled: false
aws-glue-enabled: true
# Optionally, exclude managed policies and roles matching certain patterns. Other filtering options are available too
aws-access-managed-policy-excludes: Amazon.+,AWS.+,cdk.+,AdministratorAccess,AccessAnalyzer.+
aws-access-role-excludes: AWS.+,aws.+,cdk.+,AccessAnalyzer.+
It contains
- a section to configure the connection to Raito Cloud:
api-user,api-secret, anddomain.domainis the part of the URL from your Raito Cloud instance (e.g. https://domain.raito.cloud).api-userandapi-secretare the login credentials for your Raito Cloud instance. targetshas one optional target defined for the AWS organization connector and one for AWS account connector. You can copy paste this section from the snippet that is shown on the page of the respective newly created data source in Raito cloud. The first part defines the target, connector and corresponding object ID’s in Raito Cloud (i.e.data-source-idandidentity-store-id). The second part is the configuration specific to the connectors.
Feel free to customize this configuration further. Find more information in the sections about general configuration, AWS organization-specific configuration and AWS account-specific configuration.
Remember that you can use double curly brackets to reference environment variables, like we did for the api-user field and others in the example.
Raito run
Now that our data source is set up and we have our Raito CLI configuration file, we can run the Raito CLI with:
$> raito run
This will download all data objects, users, access controls and data usage information from AWS and upload it to Raito Cloud. It will also get the access controls created in Raito Cloud and push them back into AWS (as IAM role, IAM policy, S3 Access point or Permission set), but since you’ve started out with an empty instance, this is not relevant at this point.
See here for more information about what happens exactly.
Check results in Raito Cloud
When the raito run command finished successfully, go back to Raito Cloud.
On the dashboard you will now see some initial insights that we extract from the data that was synchronized. If you go to Data Sources and visit the data sources that you have created before, you should be able to see when the last sync was done in the General information section. When you scroll down, you can also navigate through the data objects in your AWS S3 warehouse.
When you go to Identities in the navigation bar, you can see all the users and groups imported from your AWS organization and/or AWS account. Under Access Controls, under grants, you have an overview of all the access controls. If you click on one, you get a detailed view of who belongs to that access control, and what they have access to with which permissions.
Now that you have synchronized your AWS organization and the first AWS Account, you can repeat the steps to connect other AWS Accounts by creating new AWS Account data sources in Raito and configuring them using the same steps as before.
