DEV Community

Chidubem Okoli
Chidubem Okoli

Posted on

Understanding S3 Prefixes When Using Amazon Bedrock Knowledge Bases

Introduction

While working with Amazon S3 as a data source for an Amazon Bedrock Knowledge Base, I came across a concept that seems simple but is important to understand correctly:

S3 folders do not work like folders on a traditional file system.

When you open an S3 bucket in the AWS Management Console, you can create folders, upload files into them, and navigate through a familiar folder structure.

However, Amazon S3 uses a flat object storage model. What looks like a folder is generally a prefix within an object's key.

This distinction becomes particularly important when connecting an S3 bucket to an Amazon Bedrock Knowledge Base.

The prefixes configured in the Bedrock data source determine which objects are included when the Knowledge Base ingests documents.

In this article, I'll explain how S3 prefixes work, how they relate to Bedrock Knowledge Base ingestion, and how I organized documents for the Daara Media AI Creative Content Assistant.

1. The misconception: S3 folders don't really exist

When people first work with Amazon S3, they often assume that its folder structure works like the directory structure on a Linux server or Windows computer.

For example, consider this structure:

daara-media-content/
│
├── brand-guidelines/
│   └── DAARA-Media-Brand-Guidelines.pdf
│
├── past-scripts/
│   └── DAARA-Media-Script-Bank.pdf
│
├── captions/
│   └── DAARA-Media-Caption-Bank.pdf
│
└── proposals/
    └── DAARA-Media-Client-Proposal-Template.pdf
Enter fullscreen mode Exit fullscreen mode

In the S3 console, this looks like a bucket containing four folders, each holding a document.

But S3 does not store objects inside nested directories in the same way a traditional file system does.

Instead, each object has a key.

For example:

brand-guidelines/DAARA-Media-Brand-Guidelines.pdf
Enter fullscreen mode Exit fullscreen mode

The entire string is the object's key.

The forward slash is simply part of that key. S3 uses the characters before it as a prefix that applications and the console can interpret as a folder-like hierarchy.

What S3 actually stores

Conceptually, the bucket contains objects like these:

Bucket: daara-media-content

Object 1:
Key = brand-guidelines/DAARA-Media-Brand-Guidelines.pdf

Object 2:
Key = past-scripts/DAARA-Media-Script-Bank.pdf

Object 3:
Key = captions/DAARA-Media-Caption-Bank.pdf

Object 4:
Key = proposals/DAARA-Media-Client-Proposal-Template.pdf
Enter fullscreen mode Exit fullscreen mode

There is no requirement for S3 to create a physical directory called brand-guidelines before storing an object with that key.

The key itself contains the prefix.

This is why the more accurate way to describe S3 organization is through object keys and prefixes, rather than traditional folders and subdirectories.

2. Understanding an S3 object key and prefix

Let's break down one of the Daara Media object keys:

brand-guidelines/DAARA-Media-Brand-Guidelines.pdf
Enter fullscreen mode Exit fullscreen mode

It contains two useful parts:

brand-guidelines/
        |
        v
      Prefix

DAARA-Media-Brand-Guidelines.pdf
        |
        v
    Object filename
Enter fullscreen mode Exit fullscreen mode

The complete key is:

brand-guidelines/DAARA-Media-Brand-Guidelines.pdf
Enter fullscreen mode Exit fullscreen mode

The prefix is:

brand-guidelines/
Enter fullscreen mode Exit fullscreen mode

A prefix is simply the beginning portion of an object's key.

This means multiple objects can share the same prefix:

brand-guidelines/
│
├── brand-guidelines/brand-identity.pdf
├── brand-guidelines/logo-usage.pdf
└── brand-guidelines/brand-voice.pdf
Enter fullscreen mode Exit fullscreen mode

All three objects have keys beginning with brand-guidelines/.

This is how S3 allows applications to organize and retrieve related objects without requiring a traditional directory structure.

3. Why prefixes matter when using Bedrock Knowledge Bases

Understanding prefixes becomes especially important when connecting an S3 bucket to an Amazon Bedrock Knowledge Base.

A Knowledge Base needs to know which documents it should ingest.

An S3 bucket might contain hundreds or thousands of objects, but not every object necessarily belongs in a particular Knowledge Base.

For example, a company might use one bucket to store:

  • Brand guidelines.
  • Marketing documents.
  • Customer support policies.
  • Financial reports.
  • Internal administrative files.
  • Archived documents.

If the Knowledge Base is intended to answer marketing questions, ingesting every object in the bucket may not be appropriate.

This is where inclusion prefixes become useful.

An inclusion prefix tells the Bedrock S3 data source which object keys should be included during ingestion.

Instead of treating the entire bucket as the document collection, you can restrict ingestion to selected key prefixes.

Example: Daara Media document organization

For the Daara Media AI assistant, I organized the source documents using four prefixes:

Source documents in S3 bucket

daara-media-content/
│
├── brand-guidelines/
│
├── past-scripts/
│
├── captions/
│
└── proposals/
Enter fullscreen mode Exit fullscreen mode

Each prefix represented a category of company knowledge that the assistant could retrieve.

The data source needed to include these four categories rather than unrelated objects elsewhere in the bucket.

The intended ingestion scope was:

                 Amazon S3
                     |
                     v
          daara-media-content
                     |
          +----------+----------+
          |          |          |
          v          v          v
   brand-guidelines/ past-scripts/ captions/
          |
          v
      proposals/
          |
          v
    Bedrock Data Source
          |
          v
     Document Ingestion
Enter fullscreen mode Exit fullscreen mode

The important part was not simply creating these folder-like paths in S3.

It was configuring the Bedrock data source to use the corresponding prefixes.

4. Configuring inclusion prefixes

When creating an S3 data source for a Bedrock Knowledge Base, you can specify the bucket and the prefixes that should be included.

A simplified example of the S3 data source configuration looks like this:

{
  "s3Configuration": {
    "bucketArn": "arn:aws:s3:::daara-media-content",
    "inclusionPrefixes": [
      "brand-guidelines/",
      "past-scripts/",
      "captions/",
      "proposals/"
    ]
  },
  "type": "S3"
}
Enter fullscreen mode Exit fullscreen mode

The important part is:

"inclusionPrefixes": [
  "brand-guidelines/",
  "past-scripts/",
  "captions/",
  "proposals/"
]
Enter fullscreen mode Exit fullscreen mode

Each entry identifies an S3 key prefix that the data source should include.

During ingestion, Bedrock uses these filters to determine which objects are within the configured scope.

Visualizing the selection

Imagine the bucket contains these objects:

daara-media-content/
│
├── brand-guidelines/
│   └── brand-identity.pdf
│
├── past-scripts/
│   └── campaign-script.pdf
│
├── captions/
│   └── caption-examples.pdf
│
├── proposals/
│   └── proposal-template.pdf
│
└── financial-reports/
    └── annual-report.pdf
Enter fullscreen mode Exit fullscreen mode

If the inclusion prefixes are:

[
  "brand-guidelines/",
  "past-scripts/",
  "captions/",
  "proposals/"
]
Enter fullscreen mode Exit fullscreen mode

The ingestion scope becomes:

                  S3 Bucket
                      |
          +-----------+-----------+
          |                       |
          v                       v
    Included Prefixes       Not Included
          |                       |
          v                       v
  brand-guidelines/          financial-reports/
  past-scripts/                    |
  captions/                        v
  proposals/                  Not ingested
          |
          v
   Bedrock Ingestion
Enter fullscreen mode Exit fullscreen mode

The financial report is still stored in S3. It simply does not match any of the configured inclusion prefixes.

This is an important distinction: an object can exist in the bucket without being part of the Knowledge Base's ingestion scope.

5. Why the trailing slash matters

Consider these two prefixes:

brand
Enter fullscreen mode Exit fullscreen mode

and:

brand-guidelines/
Enter fullscreen mode Exit fullscreen mode

S3 prefix matching is based on the beginning of an object's key. It does not interpret the prefix as a directory name with special folder-boundary semantics.

For example, the prefix:

brand
Enter fullscreen mode Exit fullscreen mode

could match keys such as:

brand-guidelines/brand-identity.pdf
brand-assets/logo.png
brand-archive/old-guidelines.pdf
Enter fullscreen mode Exit fullscreen mode

All these keys begin with brand.

But if the intention is to include objects under the brand-guidelines/ path, specifying the trailing slash makes the intended scope clearer:

brand-guidelines/
Enter fullscreen mode Exit fullscreen mode

It matches keys beginning with that exact prefix, such as:

brand-guidelines/brand-identity.pdf
brand-guidelines/logo-usage.pdf
Enter fullscreen mode Exit fullscreen mode

It does not match:

brand-assets/logo.png
brand-archive/old-guidelines.pdf
Enter fullscreen mode Exit fullscreen mode

This is why I recommend using the full folder-like prefix, including the trailing slash, when the intention is to target a specific category.

It makes the configuration easier to understand and reduces accidental matches.

6. What happens when the prefixes are incorrect?

This is where the concept becomes practical.

Imagine uploading documents into:

brand-guidelines/
past-scripts/
captions/
proposals/
Enter fullscreen mode Exit fullscreen mode

But configuring the Bedrock data source with:

{
  "s3Configuration": {
    "bucketArn": "arn:aws:s3:::daara-media-content",
    "inclusionPrefixes": [
      "documents/"
    ]
  },
  "type": "S3"
}
Enter fullscreen mode Exit fullscreen mode

The problem is that the configured prefix does not match the object keys.

The documents exist in S3, but their keys do not begin with documents/.

As a result, the data source will not include those objects through that inclusion filter.

This can create a confusing situation:

  • The S3 bucket contains the expected PDFs.
  • The folder-like paths are visible in the console.
  • The Knowledge Base exists.
  • The ingestion job may run.
  • But the expected documents are not ingested.

The problem may not be the PDF files, the embedding model, or the Knowledge Base itself.

It may simply be an incorrect inclusion prefix.

A useful troubleshooting checklist

When documents are not appearing in a Knowledge Base, check:

  1. Is the correct S3 bucket configured?
  2. Are the object keys spelled exactly as expected?
  3. Do the inclusion prefixes match the beginning of those keys?
  4. Are there accidental differences in capitalization?
  5. Are the required S3 permissions configured for the Knowledge Base execution role?
  6. Has the data source actually been synchronized after the documents were uploaded?

S3 keys are case-sensitive, so these two prefixes are different:

brand-guidelines/
Brand-Guidelines/
Enter fullscreen mode Exit fullscreen mode

A small difference in capitalization can prevent an expected match.

7. How to inspect S3 object keys using the AWS CLI

The AWS Management Console provides a convenient visual representation of S3 objects, but the AWS CLI is useful when troubleshooting prefix problems.

To list all objects in a bucket:

aws s3api list-objects-v2 \
  --bucket daara-media-content \
  --region us-east-1 \
  --query 'Contents[*].Key' \
  --output table
Enter fullscreen mode Exit fullscreen mode

This displays the actual object keys rather than relying on the folder-like presentation in the console.

To inspect objects under a specific prefix:

aws s3api list-objects-v2 \
  --bucket daara-media-content \
  --prefix brand-guidelines/ \
  --region us-east-1 \
  --query 'Contents[*].Key' \
  --output table
Enter fullscreen mode Exit fullscreen mode

This limits the listing to keys beginning with brand-guidelines/.

You can repeat the command for the other prefixes:

aws s3api list-objects-v2 \
  --bucket daara-media-content \
  --prefix past-scripts/ \
  --region us-east-1 \
  --query 'Contents[*].Key' \
  --output table
Enter fullscreen mode Exit fullscreen mode

The objective is to confirm that the actual object keys match the prefixes configured in the Bedrock data source.

For large buckets, remember that list-objects-v2 returns a limited page of results unless pagination is used. The AWS CLI normally handles pagination automatically, but options such as --no-paginate can change that behaviour.

8. Prefixes and Knowledge Base synchronization

Uploading a document to S3 does not automatically mean that it has been indexed by the Knowledge Base.

There is an important distinction between storing a document and ingesting it.

The process looks like this:

          Upload Document
                 |
                 v
             Amazon S3
                 |
                 v
          Object Stored
                 |
                 v
        Bedrock Data Source
                 |
                 v
         Start Ingestion Job
                 |
                 v
         Check Object Scope
                 |
                 v
          Process Documents
                 |
                 v
       Update Knowledge Base
Enter fullscreen mode Exit fullscreen mode

The first synchronization processes the eligible content in the configured data source.

Subsequent synchronization jobs can detect added, updated, and deleted content and update the Knowledge Base accordingly.

This means that if a document is uploaded into an included prefix after the initial ingestion, you still need to synchronize the data source for the change to be reflected in the Knowledge Base.

For example:

Initial Ingestion
       |
       v
brand-guidelines/
       |
       v
brand-identity.pdf indexed
       |
       v
New document uploaded
       |
       v
logo-usage.pdf
       |
       v
Start another ingestion job
       |
       v
Knowledge Base updated
Enter fullscreen mode Exit fullscreen mode

The prefix defines the scope of eligible objects. Synchronization is the process that discovers and processes changes within that scope.

9. Prefixes are not the same as access control

There is another important distinction worth understanding.

An inclusion prefix controls which S3 objects a particular Bedrock data source considers for ingestion.

It is not, by itself, an IAM security boundary.

For example, configuring:

"inclusionPrefixes": [
  "brand-guidelines/"
]
Enter fullscreen mode Exit fullscreen mode

does not automatically prevent other AWS principals from reading objects under:

financial-reports/
Enter fullscreen mode Exit fullscreen mode

Access to S3 objects is controlled separately through IAM identity policies, bucket policies, and other applicable access controls.

Similarly, once documents have been ingested, access to retrieved Knowledge Base content must be considered separately.

AWS warns that synchronized content can become available to principals with the relevant bedrock:Retrieve permissions, so ingestion scope should be designed with data access in mind.

The distinction is:

Mechanism Responsibility
S3 object prefix Identifies a portion of an object's key
Bedrock inclusion prefix Defines which matching objects are included in ingestion
IAM policy Controls permissions to perform AWS actions
S3 bucket policy Controls access granted through the bucket resource policy
Bedrock retrieval permissions Controls who can retrieve Knowledge Base content

This matters when a bucket contains documents with different sensitivity levels.

A well-organized S3 bucket does not automatically provide document-level authorization.

10. The architecture in the Daara Media project

The S3 document structure was part of the wider RAG architecture.

The original company documents were stored in S3. The Bedrock Knowledge Base used the configured data source to ingest eligible documents, process them, and make their content available for retrieval.

The overall flow was:

                 Company Documents
                         |
                         v
                    Amazon S3
                         |
             +-----------+-----------+
             |           |           |
             v           v           v
       Brand Guides   Past Scripts  Captions
             |           |           |
             +-----------+-----------+
                         |
                         v
                  Proposals
                         |
                         v
              S3 Inclusion Prefixes
                         |
                         v
             Bedrock Knowledge Base
                         |
                         v
                    Ingestion
                         |
                         v
                Searchable Knowledge
                         |
                         v
                 Application Retrieval
                         |
                         v
                  Claude Sonnet 4.5
                         |
                         v
                  Draft Response
Enter fullscreen mode Exit fullscreen mode

The important point is that S3 and the Knowledge Base have different responsibilities.

Amazon S3 stores the original documents.

The Knowledge Base ingests eligible objects and prepares them for retrieval.

The application retrieves relevant information from the Knowledge Base and passes that context to the foundation model.

The prefix configuration connects the document organization in S3 to the ingestion scope of the Knowledge Base.

Conclusion

Understanding S3 prefixes is a small but important part of building a reliable RAG application with Amazon Bedrock.

What appears to be a folder in the S3 console is generally a prefix within an object's key. That prefix becomes important when configuring a Bedrock S3 data source because inclusion prefixes determine which objects are considered during ingestion.

In the Daara Media project, I used separate prefixes for brand guidelines, past scripts, captions, and proposals. This gave the Knowledge Base a clear document scope and made the source content easier to organize and maintain.

The biggest lesson is that storing documents in S3 and making them available to a Knowledge Base are two separate operations.

The objects must have the expected keys, the data source must include the correct prefixes, the Knowledge Base must have permission to access the objects, and the data source must be synchronized.

Once this distinction is understood, troubleshooting missing documents becomes much easier.

S3 stores objects using keys. Prefixes organize those keys. Bedrock uses the configured prefixes to determine which objects to ingest.

Top comments (1)

Collapse
 
aistackguide profile image
AI Stack Guide •

The prefix-as-string point is the one that costs people the most debugging time, and it bites twice in the same CLI you showed. aws s3api list-objects-v2 --prefix brand-guidelines matches brand-guidelines-old/ exactly as the Bedrock filter would, so a check command that omits the trailing slash will happily agree with a misconfigured data source.

Two things I would add, from running ingestion over mixed content:

  1. Spaces and unicode in keys. The console's "Copy S3 URI" percent-encodes them, but the object listing does not, so a prefix pasted from that URI never matches. Normalise keys on upload instead of patching per data source.

  2. Split prefixes by content type, not only by topic. If a bucket ever holds raw audio beside transcripts (meeting notes are the usual case), keep the audio prefix out of inclusionPrefixes entirely: audio chunks are useless for retrieval and just cost sync time.

On deletions, I would test rather than assume that removing an object from an included prefix drops its vectors on the next sync. Delete one object, sync, and retrieve something only that object contained.

Your point that an inclusion prefix is not an IAM boundary is the one an auditor asks about first: it is ingestion scope, not access control.