UNIVERSITAS NUSA MANDIRI Faculty of Information Technology · Informatics Study Programme (Master's)

Master of Informatics Thesis Defense

Development of a Hotel Venue Recommender System Based on Retrieval-Augmented Generation for MICE Event Planning

AuthorNuryani Mawar Putri

Student ID14240017

Study ProgrammeInformatics (Master's) · AI Concentration

SupervisorProf. Dr. Ir. Dwiza Riana, S.Si, MM, M.Kom, IPU, ASEAN.Eng

1,741

hotel venues in the knowledge base

32 prov.

131 cities/regencies · 91 hotel groups

320

pipeline executions (8 configurations × 40 queries)

Constraint Violation Rate of the best configuration

Defense Guide

Presentation Map

Eleven tabs on the left-hand side. The presentation follows the order of Chapter I through Chapter V; three supporting tabs are prepared for the question and answer session.

I Introduction

Background of the MICE industry, four identified problems, two research questions, objectives, and scope.

II Literature Review

RAG and Advanced RAG theory, four retrieval refinement techniques, evaluation metrics, prior research, three propositions.

III Methodology

Pipeline architecture, a dataset of 1,741 venues, a 2³ factorial ablation design, and the user evaluation design.

IV Results & Discussion

Ablation results for eight configurations, the Wilcoxon test, the cross-model robustness test, TAM and SUS results, and design implications.

V Conclusion

Conclusions on RQ1 and RQ2, contributions, limitations, and recommendations for further development and future research.

⚑ Supporting Tabs

Vento AI Demo screenshots & system flow.
Reference Papers 56 references, 12 available to read directly.
Q&A 130 examiner questions with their answers, searchable.
About the Researcher the author's profile, background, and publications.

The Core Message I Wish to Convey

In constrained entity retrieval, metadata filtering is more decisive than stacking query expansion techniques, and composing techniques is not monotonically beneficial. The initial hypothesis of this research was in fact rejected by its own data, and that is the scientific finding.

Research Summary

Abstract

The selection of hotel venues for MICE events in Indonesia remains manual, fragmented, and slow, while keyword-based search does not understand multi-criteria natural-language queries. This development research builds a venue recommender system based on Retrieval-Augmented Generation (RAG) over 1,741 hotel venues across 32 provinces and 131 cities/regencies.

The pipeline combines Multi-Query Retrieval, Metadata Filtering, and Hypothetical Document Embeddings, fused through Reciprocal Rank Fusion, implemented as the public chatbot Vento AI with the generator gemini-flash-latest.

An ablation study of eight configurations over 40 ground truth-annotated test queries (320 executions) shows that the full composition is not the best: the Metadata Filtering only configuration reaches a Context Precision@8 of 1.000 and a Constraint Violation Rate of 0%, outperforming both the baseline (0.181; 37.5%) and the full composition (0.241), significant under the Wilcoxon test (p<0.001). Evaluation with the Technology Acceptance Model (means of 4.18–4.42) and the System Usability Scale (69.9; above the threshold of 68) among MICE practitioners indicates good acceptance and usability.

Keywords

Retrieval-Augmented Generation Metadata Filtering Venue Recommender System Ablation Study MICE

Type of Research

This is development research with a quantitative approach, combining the construction of a software artefact with controlled empirical evaluation.

Summary of Figures

Key Findings on a Single Screen

The figures the examination committee is most likely to ask about, gathered on a single page.

Machine Side: The Ablation Study

1.000

Context Precision@8 of configuration C3 (MF only), also perfect on Recommendation P@5, Hit Rate, and MRR

Constraint Violation Rate of C3; no recommendation violates a constraint

0.241

Context P@8 of the full composition C8. CVR 37.5%, behind C3 and C7

p<0.001

Wilcoxon test: the advantage of C3 & C7 over C8 is statistically significant

Human Side: User Evaluation

4.39

Perceived Usefulness (5-point scale)

4.42

Perceived Ease of Use (5-point scale)

4.18

Intention to Use · Attitude 4.24

69.9

SUS score (median 70.0), above the normative threshold of 68

Data & Instrument Quality

0.900

Gradient Boosting accuracy on the learnability check (majority baseline 51.6%)

100%

Completeness of capacity & fullday price across 1,741 rows

0.966

Cronbach's α of the TAM instrument (10/10 items valid)

83.9%

Respondents with more than 5 years of work experience (26 of 31)

Notes on the Findings

Proposition P1 is not supported: the data show that the full composition is not the best, and that result is reported as it stands. The SUS score sits 1.9 points above the normative threshold with 18 valid responses, so additional respondents are needed to firm up the figure. Both are stated in Chapter IV and Chapter V as limitations of the study.

CHAPTER I · Introduction

Background

The MICE industry is high in value and is undergoing digital transformation, yet its most decisive process (venue selection) is still carried out manually.

IDR 11.82 trillion

MICE economic turnover up to September 2025

134

MICE events recorded in the same period

95 thousand

workers absorbed by the MICE industry

#38

Bali's rank as an international meetings destination (ICCA)

Why Hotel Venues Are Central

They provide meeting space and overnight accommodation in a single location, which makes their selection the most decisive stage.
The decision is multi-criteria: accessibility, venue quality, destination context, and accommodation availability.
The industry is entering a phase of digital transformation marked by rising adoption of technology and artificial intelligence, a tendency referred to as MICE 5.0.

Conditions in the Field

Organisers contact hotels one by one, request quotations, then compare prices, capacities, and facilities without any standardised data source.
No aggregation channel is yet available for the Indonesian market that presents meeting package and accommodation data in structured form according to the halfday, fullday, and fullboard schemes.
MICE pricing is dynamic and not published; reservation systems across hotel groups are siloed and not interconnected.

The Core Problem

Organisers have no independently accessible reference for preparing an initial budget estimate, even though what is actually needed at the planning stage is only an indicative price range, not a final negotiated price.

CHAPTER I · Introduction

Why Existing Technologies Fall Short

✕ Keyword search & rigid filters

Unable to answer needs that are narrative and multi-criteria. Users write sentences; they do not fill in filter boxes.

✕ Collaborative filtering

Relies on dense historical rating data; severely vulnerable to sparsity and cold start in the hospitality domain, and not designed for constraint-based requests.

✕ LLMs without grounding

They unlock the ability to understand natural language, but carry a hallucination risk that is dangerous in venue recommendation.

The Gap This Research Fills

A number of studies have applied RAG to the hospitality and tourism domains. However, to the best of my knowledge, none has applied it to hotel venue recommendation in a business-to-business MICE context, on an Indonesian-language corpus, at a scale of thousands of entities, and evaluated it both in terms of system quality and of user acceptance.

RAG in hospitalitystops at Naive RAG for a single property

Tourism recommendationleisure segment, non-generative / B2C

LLMs for RecSyshas not reached the MICE venue domain

→

GAPAdvanced RAG for Indonesian B2B MICE venue entities

CHAPTER I · Section 1.2

Four Identified Problems

The process of selecting hotel venues for MICE purposes is still manual and fragmented, and no aggregation channel is yet available for the Indonesian market that presents meeting package and accommodation data in structured form according to the customary package schemes (halfday, fullday, fullboard).
Information on MICE package prices and availability is not directly accessible to the public, so event organisers have no independent reference for preparing budget estimates and initial quotations at the planning stage.
Keyword-based search systems with rigid filters and collaborative filtering approaches cannot handle multi-criteria natural-language queries that contain hard constraints (capacity, location, budget), which is the character of MICE venue requests.
Using an LLM directly without grounding in a knowledge base carries a hallucination risk, while the effectiveness of composing retrieval refinement techniques within the Advanced RAG framework for Indonesian-language venue entity retrieval tasks has not been tested empirically.

CHAPTER I · Section 1.2

Two Research Questions

RQ1 Main Question

Which composition of retrieval refinement techniques, among the combinations of MQR, MF, and HyDE fused through RRF, produces the best hotel venue recommendation quality within the Advanced RAG framework, as measured by objective ground truth-based metrics (Context Precision@8, Recommendation Precision@5, Hit Rate, MRR) and the Constraint Violation Rate, through an eight-configuration ablation study?

RQ2 Prototype Verification

Is the Vento AI prototype, built with the best composition identified in RQ1, verified as suitable for MICE practitioners, considered in terms of technology acceptance (the perceived usefulness, perceived ease of use, attitude, and intention to use constructs of the TAM) and usability (SUS against the normative average)?

The Relationship Between RQ1 and RQ2: Worth Stating Clearly

RQ1 is the main question and forms the heart of the scientific contribution; its answer is obtained through a quantitative ablation study and stands on its own as a knowledge claim. RQ2 is not a separate line of enquiry but a direct derivative of RQ1; its function is limited to verifying whether the pipeline whose composition was tested in RQ1 is also accepted and usable by the target practitioners once it is realised as a prototype. Together they form a single chain of triangulated evaluation rooted in RQ1.

Infographic of the research questions — **Visual summary of the research questions.** RQ1 selects the best pipeline → realised as the Vento AI prototype → RQ2 verifies its acceptance and usability.

CHAPTER I · Section 1.3

Research Objectives

General objective: to develop and evaluate a query-based hotel venue recommender system using an Advanced RAG approach in support of MICE event planning in Indonesia.

Specific Objectives

1 Build

A hotel venue recommender system prototype (Vento AI) on a knowledge base of 1,741 venues, with a retrieval pipeline that composes MQR, Metadata Filtering, and HyDE through RRF fusion.

2 Measure (RQ1)

The contribution of each retrieval refinement technique to recommendation quality through an ablation study with objective ground truth-based metrics and CVR.

3 Measure (RQ2)

User acceptance and system usability among respondents who are event organisers and related practitioners, using the TAM and SUS instruments.

Expected Benefits

Theoretical

Empirical evidence on the effectiveness of composing retrieval refinement techniques for constrained entity retrieval tasks in the Indonesian-language B2B MICE domain, a configuration not yet tested in previous literature.

Practical

An independent indicative reference for event organisers in preparing budget estimates and initial quotations; for hotels, an additional visibility channel for their MICE packages.

Academic

The prototype together with a set of 40 test queries with ground truth that can serve as material for further study in the field of Indonesian-language RAG.

CHAPTER I · Section 1.4

Research Scope & Limitations

The limitations are stated openly from the outset so that the claims of this research remain proportionate.

Recommendation object hotel and resort venues in Indonesia that offer meeting packages, because their package structure is integrated and relatively uniform and can therefore be standardised.
Knowledge base 1,741 venues across 32 provinces and 131 cities/regencies, frozen as a snapshot; prices are positioned as an indicative reference, not real-time negotiated prices.
Language the system accepts queries and produces recommendations together with justifications in Indonesian.

Intervention limited to the retrieval stage at inference time (MQR, MF, HyDE, RRF fusion, context limiting and ordering). There is no fine-tuning and no pre-training; the LLM is accessed as a black-box service.
Evaluation covers (a) a quantitative ablation study with objective metrics and CVR, and (b) user evaluation with TAM and SUS.
Output a web chatbot prototype. It does not cover transactions, bookings, real-time availability, or dynamic pricing, which remain the domain of the hotels themselves.

Research Assumptions

The curated dataset is assumed to be sufficiently representative of the Indonesian MICE hotel venue market within the segment it serves. Consequently, the claims of this research are limited to that knowledge base corpus and are not generalised to the population of all hotels in Indonesia.

CHAPTER II · Literature Review

Map of the Theoretical Foundation

Nine theoretical pillars supporting this research, from the industry domain through to the evaluation instruments.

2.1.1 The MICE Industry

MICE as a part of event tourism; site selection is a multi-criteria decision. Contemporary destination assessment frameworks condense it into four factors: accessibility, venue, destination context, accommodation.

2.1.2 Recommender Systems

From content-based and collaborative filtering methods and hybrids towards conversational recommender systems. This research sits in the class of zero-shot style, query-driven recommendation, with session-level personalisation rather than historical profiles.

2.1.3 Dense Retrieval

Sparse approaches (TF-IDF, BM25) fail when vocabularies differ. Dense retrieval represents queries and documents as dense vectors; relevance is measured by cosine similarity.

2.1.4 Sentence Embeddings

Sentence-BERT trains a siamese network on top of BERT so that sentence embeddings can be compared directly, reducing the cost of comparison from the order of hours to milliseconds.

2.1.5 The LLM as Generator

Two relevant limitations: hallucination (intrinsic/extrinsic in the taxonomy of Ji et al.) and the knowledge cutoff. Both are reasons why RAG is needed.

2.1.6 RAG

Combines a retriever and a generator: p(y | x, d) instead of p(y | x). The taxonomy of Gao et al.: Naive → Advanced → Modular RAG.

2.1.7 Refinement Techniques

MQR, Metadata Filtering, HyDE, RRF, together with context limiting and ordering based on the Lost in the Middle finding.

2.1.8 Vector Databases

They store embeddings and provide efficient nearest-neighbour search through ANN indexes, trading a little accuracy for a multifold gain in speed.

2.1.9 System Evaluation

Objective metrics based on ground truth + CVR for the machine side; TAM and SUS for the human side.

CHAPTER II · Section 2.1.6

Retrieval-Augmented Generation

RAG combines a retriever component and a generator component. On receiving a query x, the system first retrieves the relevant documents d from the knowledge base, after which the generator produces the answer y conditioned on both.

p(y | x, d) instead of p(y | x)

This mechanism suppresses the risk of hallucination because the answer is grounded in factual data that can be updated without retraining the model.

The Taxonomy of Gao et al.

Naive RAG a direct retrieve-and-read flow.
Advanced RAG adds pre-retrieval optimisation (query transformation, metadata filtering) and post-retrieval optimisation (fusion, re-ranking, context compression). ← the position of this research
Modular RAG arranges the components flexibly.

Why the Offline–Online Separation Matters

Every intervention tested lies in the online stage (at inference time), whereas the vector index built in the offline indexing stage is kept identical for all eight ablation configurations. In this way, differences in the results cannot possibly originate in differences between indexes.

**Figure II.1** The general RAG architecture and the Advanced RAG optimisation points. The dashed boxes are optimisation points that operate at inference time without retraining the model. *Source: prepared by the author (2026)*

CHAPTER II · Section 2.1.7

The Four Techniques Being Composed

Each technique has been validated separately in its primary literature. What this research tests is their composition.

1 Multi-Query Retrieval (MQR)

It generates several variants of a single user query in order to capture a diversity of intents and terms, runs a search for each variant, then merges the results so that coverage of the relevant documents increases.

Foundation: LLM-based query transformation within the rewrite-retrieve-read paradigm; a user's original query is not the optimal form for the retriever. The multi-query pattern combined with rank fusion has been popularised as RAG-Fusion.

2 Metadata Filtering (MF)

It filters candidate venues by structured attributes (city, star rating, capacity, price range) before or alongside the semantic search.

Its role is decisive: semantic search is approximate, so it can let through candidates that are similar in meaning yet violate the numerical requirements. MF closes that gap and keeps the recommendations inside the space of valid options.

3 Hypothetical Document Embeddings (HyDE)

It generates a hypothetical document representing the ideal answer to a query, then uses the embedding of that document instead of the query embedding in order to find the most similar real documents.

Theoretically: it converts the query-document matching problem into document-document matching, whose embedding distributions are more comparable.

4 Reciprocal Rank Fusion (RRF)

RRF(d) = Σ_r∈R 1 / (k + rank_r(d))

It merges several ranked lists into a single final ranking. The constant k = 60 dampens the dominance of any single top rank. For example, a venue ranked 1 in two lists obtains 1/61 + 1/61 = 0.0328, outranking a venue that appears in only one list (0.0164); consensus across retrieval paths wins without manual weighting.

Context Limiting & Ordering, Lost in the Middle

Liu et al. show that LLM performance forms a U-shaped curve with respect to the position of information within a long context: information at the beginning and at the end is used well, whereas information in the middle tends to be neglected. This finding underpins two design decisions: (1) the fused context is limited to the eight best candidates, and (2) the highest-scoring candidates are placed at the beginning of the context.

CHAPTER II · Section 2.1.9

The Evaluation Framework: Machine and Human

Five Objective Metrics (Machine Side)

Context Precision@8

The proportion of relevant documents among the top eight contexts retrieved. Range 0–1; the higher, the better.

Recommendation Precision@5

The proportion of final recommendations that are correct. Range 0–1.

Hit Rate & MRR

Hit Rate: the proportion of queries with at least one correct recommendation. MRR: the mean reciprocal rank of the first correct recommendation.

CVR, formulated by the author

The percentage of queries whose recommendations violate at least one constraint. The lower, the better.

CVR = (|Q_violating| / |Q|) × 100%

where Q is the set of all test queries and Q_violating is the set of queries whose recommendations violate the location, capacity, star rating, or budget constraints.

Why Deterministic Metrics Rather Than an LLM Judge?

Automated LLM-based evaluation frameworks such as RAGAS are certainly available for open-topic RAG pipelines. On a constrained task whose set of correct answers is well defined, however, deterministic metrics are more objective, exactly reproducible, and free of the cost and the variability of a judge. RAGAS is still used, but as a cross-model robustness test (Section 4.2.1), not as the primary metric.

Human-Side Instruments

Technology Acceptance Model (TAM)

It explains technology acceptance through perceived usefulness and perceived ease of use, which influence attitude and culminate in behavioural intention to use. The items were adapted from Davis's original instrument using a five-point Likert scale.

System Usability Scale (SUS)

SUS = 2.5 × [ Σ odd (s − 1) + Σ even (5 − s) ]

Ten items on a five-point Likert scale, yielding a range of 0–100. The empirical reference of Bangor et al.: above 68 lies above the industry average, while 80 and above counts as excellent.

CHAPTER II · Section 2.2

Synthesis of Prior Research

The review is organised into five groups, each closing with the position of this research relative to the studies concerned.

Study	Method	Dataset / Context	Results & Limitations	Difference from This Research
Wijaya & Jayadianti [16]	Naive RAG (Qdrant, chunking 1,000/200)	1 Indonesian hotel; 15 test cases	Good RAGAS scores (CP 0.889; F 0.882); no advanced retrieval techniques, no user evaluation	Advanced RAG with 4 techniques; 1,741 venues; ablation + TAM + SUS
Karlović et al. [17]	RAG + semantic re-ranking; 7 local LLMs	Lošinj tourism brochures; synthetic profiles	Re-ranking is effective; the choice of LLM matters; leisure segment	B2B MICE domain; real users; constrained entities
Banerjee et al. [18]	RAG + re-ranking based on sustainability metrics	Tourism destinations (leisure)	Domain criteria inserted during augmentation; no hard numerical constraints	Numerical constraints guaranteed by MF; B2B MICE venue domain
Song et al. [19]	TravelRAG: multi-layer knowledge graph	Social media UGC	Retrieval accuracy improves; KG construction is complex & costly	No KG; a lightweight composition of inference-time techniques
Shambour et al. [7]	FBMCCF (fused multi-criteria CF)	TripAdvisor: 28,829 ratings, 693 hotels	Outperforms the baselines (MAE/RMSE/coverage); requires a dense rating matrix	No rating history; constraint-based retrieval + generation
Wei et al. [20]	ER2ALM: RAG + LLM for RecSys	2 recommendation datasets	Addresses cold start & sparsity; general e-commerce domain	MICE venue domain; B2B natural-language queries
Setiyawan & Kenoya [8]	FAQ chatbot: Random Forest + Word2Vec	ICT service FAQ (Telegram)	Accuracy 91.28%; answers only from a closed list, requires retraining	Generative and grounded in data; no retraining
Putri & Saputra [10]	Graph-based event vendor recommendation	Indonesian event vendors	Inter-vendor relations are modelled; no generative natural-language interface	Natural-language queries + grounded generative answers; research continuity
Suryadi et al. [13]	Automatic QA dataset (privacy-preserving) for refining LLMs	Internal documents; local models	A QA-IR pipeline for fine-tuning local LLMs; not a recommender system	A black-box LLM without fine-tuning; venue entity retrieval
This research (Putri, 2026)	Advanced RAG: MQR + Metadata Filtering + HyDE, RRF fusion; inference-time	1,741 Indonesian hotel venues (32 provinces, 131 cities/regencies)	Ablation study (ground truth + CVR) and user evaluation (TAM + SUS)	Novelty: the composition of four retrieval techniques for Indonesian B2B MICE venues, doubly evaluated

Table II.1 Synthesis of Prior Research, Source: prepared by the author from [7], [8], [10], [13], [16]–[20]

CHAPTER II · Sections 2.2 & 2.3

Synthesis of the Gap and the Conceptual Framework

Three Conclusions from the Review

All the techniques making up the pipeline in this research have been validated separately in their primary literature, and some of their combinations (multi-query with RRF) are already acknowledged as effective practice.
The application of RAG in hospitality and tourism still stops at Naive RAG on a single property or at destination recommendation in the leisure segment.
Non-generative approaches prove unsuited to the structure of the B2B venue procurement problem, which is constraint-based and has no interaction history.

The Gap Being Filled

No research has yet been found that composes MQR, MF, HyDE, and RRF within a single Advanced RAG pipeline for the retrieval of Indonesian-language venue entities at a scale of thousands of entities, tested through an ablation study and a user evaluation at once.

The Three Propositions Tested

P1 Tested through ablation

The full composition of retrieval refinement techniques (MQR + MF + HyDE + RRF) yields better recommendation quality, higher precision and lower constraint violation, than configurations that disable any one of those techniques.

P2 Tested through TAM

Perceived usefulness and perceived ease of use relate positively to users' intention to use the system.

P3 Tested through SUS

The system's usability score lies above the SUS normative average threshold.

A Preview of the Results: To Anticipate Questions

In Chapter IV, P1 will prove NOT to be supported. P2 and P3 are supported, with a note on the narrow SUS margin. A study that reports the rejection of its hypothesis is precisely one that shows its testing was genuine rather than confirmatory.

CHAPTER III · Methodology

Research Stages

Development research with a quantitative approach, combining the construction of a software artefact with a controlled empirical evaluation.

Literature study and identification of the research gap
Preparation of the knowledge base
Prototype development
Ablation experiment
User evaluation
Data analysis and discussion
Drawing of conclusions

Two Complementary Evaluation Paths

The ablation experiment (3.4) objectively measures the contribution of each technique to machine quality over test queries that have a ground truth. The user evaluation (3.5) assesses the acceptance and the usability of the working prototype from the human side.

The first path answers 'which composition of techniques is the most accurate in machine terms'; the second answers 'whether the system is accepted and easy to use by practitioners', so that the machine conclusions and the human conclusions do not substitute for one another.

Research framework — **Figure III.1** Research Framework. *Source: prepared by the author (2026)*

CHAPTER III · Section 3.2

Vento AI System Design

The principal actor is a general one: the user as an event planner, covering MICE practitioners as well as event organisers more broadly, who enters the event requirements in natural language and then receives venue recommendations together with their justifications.

Use Case Extensions

«extend» Viewing venue details (capacity, packages, indicative prices)
«extend» Refining the criteria and searching again
«extend» Comparing venue alternatives
«include» Receiving recommendations always begins with entering the requirements

Vento AI system use case diagram — **Figure III.2** Use Case Diagram of the Vento AI System. *Source: prepared by the author (2026)*

CHAPTER III · Section 3.2

Vento AI Pipeline Architecture

Indonesian-language queries are processed through three parallel retrieval paths, all of which read one and the same vector index containing 1,741 venues.

User QueryIndonesian, free-form

→

Query Parserreads city, capacity, budget, package

→

Embedding Encodermultilingual MiniLM, 384 dim

MQR Path3 query variants

MF Pathhard constraint filter

HyDE Pathhypothetical document

ChromaDBvector index, cosine/ANN

→

RRF (k=60)fusion of 3 ranked lists

→

Top-8 ContextLost in the Middle arrangement

Generatorgemini-flash-latest · temp 0.2

→

Recommendations + Justificationtop 5 venues + detail cards

Critical Decision Point at the Output Stage

If the results satisfy all hard constraints (city, capacity, budget, star rating), the system displays the top five venues together with their detail cards. If no venue satisfies them, the system neither forces an answer nor fabricates venues; instead it states that limitation honestly, together with a reasonable price reference and suggestions for adjusting the criteria, while still showing the closest available venues. This behaviour is one of the factors that keeps the CVR low.

Vento AI pipeline architecture — **Figure III.3** Vento AI Pipeline Architecture, from the presentation layer to the business layer. *Source: prepared by the author (2026)*

CHAPTER III · Section 3.2

Trace of a Single Query (Worked Example)

Test Query Q05

"Find a hotel venue in Bandung for a full-day meeting, around 300 participants, budget ≤ IDR 450,000/pax."

MQR, query paraphrasing

1. "hotels in Bandung with a meeting room for 300 people, full day, affordable"
2. "full-day meeting venues in Bandung for 300 participants under IDR 450k/person"
3. "corporate event spaces in Bandung for 300 pax, one-day meeting package"

Each variant is searched separately, then fused. Temperature 0.7.

MF, mandatory constraint filter

city = "Bandung"
capacity ≥ 300
package = full-day
fullday_price ≤ 450,000

Only venues that pass this filter are retrieved; this is the constraint guarantor.

HyDE, hypothetical venue

The LLM composes an ideal document: "Grand Serela Bandung, a 4-star hotel in the city centre, a ballroom for 350 pax, a full-day package at IDR 420,000/pax including two coffee breaks and lunch, complete AV equipment, ample parking…"

The embedding of this document attracts similar venues. Temperature 0.5, ±80 words.

Reciprocal Rank Fusioncombines the three ranked lists, k = 60

→

Top-8 Context → Generatorgemini-flash-latest, grounded in the retrieved venues

→

RecommendationsGrand Serela Bandung IDR 420k/pax, 350 pax · Holiday Inn Bandung IDR 445k/pax, 320 pax

Query trace example on the Vento AI pipeline — **Figure III.4** Query Trace Example (Worked Example) on the Vento AI Pipeline. Each recommendation satisfies the city, capacity, package, and budget constraints. *Source: prepared by the author (2026)*

CHAPTER III · Section 3.2

Selection of Technology and Operational Parameters

Three considerations: alignment with the research design (black-box, inference-time), support for the Indonesian language, and affordable computational cost.

Component	Parameter	Value
Each retrieval path	Candidates retrieved	Top 20
RRF fusion	Constant k	60
Context assembly	Documents passed to the generator	Top 8 (Lost in the Middle arrangement)
MQR	Number of query variants; temperature	3 variants; 0.7
HyDE	Number of hypothetical documents; temperature	1 document; 0.5
Generator	Model; temperature	`gemini-flash-latest`; 0.2
Embedding	Model; dimensions	paraphrase-multilingual-MiniLM-L12-v2; 384
Vector database	Product	ChromaDB (embedded mode)

Why Gemini Flash?

It supports the Indonesian language well, its latency and cost are viable for an interactive chatbot, and it represents the black-box class of LLMs assumed by the design, in line with the methodological position of REPLUG.

Why MiniLM with 384 dimensions?

A multilingual Sentence-BERT model that supports Indonesian, light enough to run without a GPU, and already widely used across languages.

Why ChromaDB?

It provides built-in metadata filtering, a core requirement for the MF component, with an embedded mode that simplifies integration. Functionally equivalent to Qdrant, which was used by Wijaya & Jayadianti.

Table III.1 System Operational Parameters, all of which are kept identical across the ablation configurations, so that differences in results can be attributed solely to the technique under test. Source: prepared by the author (2026)

CHAPTER III · Section 3.3

Research Data: 1,741 Hotel Venues

1,741

hotel venues

provinces

131

cities/regencies

hotel groups

The knowledge base was compiled by the author from domain knowledge built over more than 14 years of practice in the MICE industry together with searches of public sources. Each venue has identity attributes (name, hotel group, star rating), location (province, city/regency, address), meeting room capacity, seven price attributes covering meeting packages and accommodation, facility attributes, and a description. The dataset was frozen as a snapshot so that all experiments and evaluations read identical data.

Seven Price Attributes Mapped to MICE Service Schemes

Dataset Attribute	Service Scheme	Unit	Description
`halfday_meeting_pax`	Non-residential meeting	per pax	±4 hours, 1× coffee break
`fullday_meeting_pax`	Non-residential meeting	per pax	±8 hours, 1× meal + 2× coffee breaks, used as the reference price for budget filtering
`fullboard_meeting_pax`	Non-residential meeting	per pax	±12 hours, 2× meals + 2× coffee breaks
`single_residential_pax`	Fullboard residential	per pax	overnight stay + meeting, 1 pax/room
`twin_residential_pax`	Fullboard residential	per pax	overnight stay + meeting, 2 pax/room
`triple_residential_pax`	Fullboard residential	per pax	overnight stay + meeting, 3 pax/room
`extrabed_pax`	Fullboard residential	per pax	additional bed (quadruple occupancy)

Table III.2 Mapping of Dataset Price Attributes to MICE Service Schemes. The fullday price was chosen as the reference because it is the option most commonly selected in corporate event planning and also the only price attribute with 100% completeness. Source: prepared by the author (2026)

CHAPTER III · Sections 3.3.2 & 3.3.3

Preprocessing and Data Learnability Check

Preprocessing

Venue type attribute: 21 mixed categories + 311 empty entries (17.9%) → standardised into 6 standard categories (Hotel, Resort, Serviced Residence, Inn/Lodge, Villa, Non-Hotel Venue), with empty entries grouped as "Uncategorised".
Concept tag attribute: 54 mixed values → 6 standard categories (Business/MICE, Premium, Budget, Leisure/Resort, Boutique/Lifestyle, General Hotel).
Final curation: duplicate entries across sources were removed; invalid prices (below IDR 50,000/pax) were imputed with the median of the brand and star-rating class taken from verified rows.
Every row was given a provenance marker (scraped/curated) so that all steps can be audited.

Data Learnability Check (Learnability Check)

Before being put to use as the knowledge base, the dataset was checked for learnability: two baseline algorithms, Random Forest and Gradient Boosting, were trained to predict a venue's star rating from five non-price features (capacity, number of facilities, type, concept tag, province), with a stratified 80:20 train-test split and five-fold cross-validation.

Important to Emphasise

This diagnostic check sits outside the recommendation pipeline. Random Forest and Gradient Boosting are not used to recommend anything; their sole function is to examine whether the structured attributes are consistent and carry signal, and are therefore suitable for use as filtering metadata.

CHAPTER III · Section 3.4

Ablation Experiment Design: Answering RQ1

A full factorial design of 2 × 2 × 2 = 2³ = 8 configurations. Each of the three techniques can be enabled or disabled.

Configuration	MQR	MF	HyDE	Description
C1	–	–	–	Pure semantic retrieval, the equivalent of Naive RAG (baseline)
C2	✔	–	–	MQR only
C3	–	✔	–	Metadata Filtering only
C4	–	–	✔	HyDE only
C5	✔	✔	–	MQR + MF
C6	✔	–	✔	MQR + HyDE (in the same family as RAG-Fusion)
C7	–	✔	✔	MF + HyDE
C8	✔	✔	✔	The complete Vento AI pipeline, hypothesised to be the best (P1)

Ablation study configurations C1-C8 — **Figure III.5** Ablation Study Configurations (C1–C8) and the identical flow followed by every configuration. *Source: prepared by the author (2026)*

Test Query Set

40 Indonesian-language queries constructed from real request patterns in the MICE industry.
Covering variation across several dimensions: large cities and small cities, capacities from small up to thousands of participants, all three package schemes, with and without a budget limit, with and without accommodation requirements, as well as several difficult queries whose constraints are met by only a few venues.
Each query is accompanied by a ground truth in the form of a list of venues satisfying all of its criteria, verified directly against the structured data.

320 Pipeline Executions

Every configuration answers the same 40 queries in an identical order and with identical parameters, producing 8 × 40 = 320 pipeline executions. The score per configuration is the average over the 40 queries.

For significance testing, each query is summarised into a single composite score combining the five metrics, and the differences between configurations are then tested with the Wilcoxon signed-rank test, the nonparametric counterpart of the paired t-test, appropriate here because the design is paired and the distribution cannot be assumed to be normal.

CHAPTER III · Section 3.5

User Evaluation Design: Answering RQ2

Population & Sample

Target population: practitioners in Indonesia's event ecosystem. The sample was drawn purposively with a target of at least 30 valid respondents. Inclusion and exclusion criteria were set before the analysis:

Respondents work within the event ecosystem: owners, staff, or freelancers of event organizer businesses, vendors/partners, corporate tour guides, or representatives of companies that use EO services.
All closed-ended questionnaire items are fully completed.
The answer pattern passes the consistency check; it does not assign uniformly high (or low) scores to positively worded and negatively worded items alike.

The procedure: respondents use Vento AI directly for a real task scenario, searching for a venue matching the needs of an event they are currently handling or have handled in the past, and then complete an online questionnaire.

Instrument Blueprint

Construct	Items	Example Item
Perceived Usefulness	3	"Vento AI helps me find venues that match my needs more quickly"
Perceived Ease of Use	3	"Vento AI is easy for me to use"
Attitude Toward Use	2	"Using Vento AI is a good idea for my work"
Intention to Use	2	"I intend to use Vento AI again in the future"
Usability (SUS)	10	"I think I would use Vento AI frequently" (item 1 of 10)
Open feedback	1	Suggestions, feedback, or notes for further development

Table III.4 Instrument Blueprint for the User Evaluation. The TAM items are adapted from Davis [26]; the SUS items are an Indonesian translation that preserves the original item order and polarity of Brooke [44].

CHAPTER III · Section 3.5.3 & 3.5.4

Instrument Quality Testing and Analysis Methods

Instrument / Construct	Items	Cronbach α	Valid Items	Notes
TAM: Perceived Usefulness	3	0.932	3/3	Valid and reliable
TAM: Perceived Ease of Use	3	0.857	3/3	Valid and reliable
TAM: Attitude	2	0.875	2/2	Valid and reliable
TAM: Intention to Use	2	0.915	2/2	Valid and reliable
TAM (combined)	10	0.966	10/10	Highly reliable (r ranges from 0.695–0.918)
SUS	10	0.757	7/10	Reliable; the 3 standard items were retained

Table III.5 Summary of the Instrument Validity and Reliability Testing. Testing covered all responses with complete answers: TAM n = 31; SUS n = 29. r-table for TAM (df=29) 0.355; for SUS (df=27) 0.367. Source: primary data processed by the author (2026)

Why Were the 3 Invalid SUS Items Retained?

Because SUS is a standardised international instrument whose item order and polarity are deliberately designed as an intact whole so that its score can be compared against industry norms. Dropping items in order to raise validity would make the final score no longer comparable with the threshold of 68, which is precisely the core of the P3 test. Overall reliability remains adequate (0.757 > 0.70).

Data Analysis Methods

TAM: analysed descriptively (mean per construct); the relationships between constructs for P2 were tested with Spearman correlation (PU→intention, PEOU→intention) at the 5% level, nonparametric because the data are ordinal.

SUS: each respondent's score was computed using the standard scoring rules, and the mean was then compared against the normative threshold of 68 to test P3, accompanied by reporting of the median, the distribution, and the proportion of respondents above the threshold.

Open-ended questions: analysed through simple thematic analysis to enrich the discussion.

CHAPTER IV · Section 4.1.1

Implementation Results: Dataset Characteristics

Aspect	Description
Data source	Public catalogue of a venue reservation platform (data dictionary in the Appendix)
Number of entities	1,741 hotel venues
Geographic coverage	32 provinces; 131 cities/regencies; 91 hotel groups
Structured attributes	City, province, group, maximum capacity, halfday/fullday/fullboard price per pax, extrabed
Text attributes	Venue and facility descriptions, converted into 384-dimensional multilingual MiniLM embeddings
Role in the system	Retrieval knowledge base; no model training
Pre-processing	Normalisation, deduplication, median price imputation, package schema mapping

Table IV.1 Characteristics of the Knowledge Base Dataset. Source: processed by the author (2026)

Data Completeness

100%

capacity & fullday price populated across all 1,741 rows

100%

halfday & fullboard prices (documented imputation results)

99.4%

facility attributes populated

3.8%

public ratings, too sparse, not used

Implication of the Absence of Ratings

Public ratings are available for only 3.8% of the rows. This is at the same time the structural reason why a rating-based approach (collaborative filtering) is not viable in this domain, rather than merely a methodological preference.

CHAPTER IV · Section 4.1.2

Data Learnability Check Results

0.883

Random Forest test accuracy
Macro F1 0.784 · 5-fold CV 0.892 ± 0.007

0.900

Gradient Boosting test accuracy
Macro F1 0.798 · 5-fold CV 0.907 ± 0.014

51.6%

majority-class baseline; both models outperform it by a wide margin

These results show that the non-price structured attributes are consistent and informative, so they are suitable for use as filtering metadata and as material for representing venue profiles in the RAG pipeline.

Once Again: This Is Not Part of the Pipeline

Random Forest and Gradient Boosting do not recommend anything. This test is purely diagnostic, confirming that the data are not random before they are used as a knowledge base.

Code and output of the learnability check experiment — **Figure IV.1** Code and Output of the Learnability Check Experiment. *Source: screenshot of the author's experiment notebook (2026)*

CHAPTER IV · Section 4.2 · ANSWERING RQ1

Ablation Experiment Results

The experiment was run through the Vento_AI_Ablasi_KOKOH notebook on Google Colab for a total of 320 runs (8 configurations × 40 test queries), with the operational parameters kept identical.

Configuration	Context P@8	Rec P@5	Hit Rate	MRR	CVR (%)	p (Wilcoxon vs C8)
C1 (baseline · Naive RAG)	0.181	0.667	0.700	0.688	37.5	0.004
C2 (MQR)	0.162	0.658	0.700	0.675	37.5	<0.001
C3 (MF): BEST	1.000	1.000	1.000	1.000	0.0	<0.001
C4 (HyDE)	0.116	0.492	0.500	0.488	52.5	<0.001
C5 (MQR + MF)	0.256	0.783	0.825	0.812	27.5	0.272
C6 (MQR + HyDE)	0.169	0.669	0.725	0.712	37.5	0.001
C7 (MF + HyDE): SECOND	0.522	0.978	1.000	0.962	7.5	<0.001
C8 (full) (hypothesised to be the best)	0.241	0.715	0.775	0.762	37.5	-

Table IV.2 Ablation Experiment Results (averaged over 40 test queries). The first four columns range from 0–1 (higher is better); CVR is expressed in percent (lower is better). The final column reports the p values of paired Wilcoxon tests on the composite score against C8 at the 5% level. Source: the author's experimental results (2026)

Context Precision@8, higher is better

Ranking: C3 (1.000) > C7 (0.522) > C5 (0.256) > C8 (0.241) > C1 (0.181) > C6 (0.169) > C2 (0.162) > C4 (0.116)

Best MF-bearing configurations (C3, C7)C8, full composition (reference)Other configurations

Constraint Violation Rate (%), lower is better

CVR rises as more retrieval paths bypass the metadata filter

Constraint-compliant (CVR ≤ 7.5%)C8, full compositionOther configurations

Code and output of the ablation experiment notebook — **Figure IV.4** Code and Output of the Ablation Experiment Notebook: the 2³ factorial matrix of eight configurations, the *retrieval* function with RRF fusion, and the metric summary output. *Source: screenshot of the author's experiment notebook (2026)*

CHAPTER IV · Section 4.2 · ANSWERING RQ1

Three Main Findings of the Ablation Study

A Finding That Runs Counter to the Hypothesis

The full configuration C8, hypothesised to be the best, in fact occupies a middle position, whereas the best configuration is C3, which activates Metadata Filtering only, with a perfect score on every precision metric (Context P@8, Rec P@5, Hit Rate, and MRR all 1.000) and a CVR of 0%.

1 Dominance of Metadata Filtering

All four configurations containing MF (C3, C5, C7, C8) consistently yield a higher Context P@8 and a lower CVR than the four configurations without MF (C1, C2, C4, C6).

Context P@8 range of the MF-bearing group: 0.241–1.000. The group without MF: only 0.116–0.181.

This is consistent with the argument in 2.1.7: MF filters directly on structured attributes and therefore ensures that candidates genuinely satisfy the criteria, something that semantic similarity alone cannot guarantee.

2 Adding expansion on top of MF actually lowers precision

Among the MF-bearing configurations: pure MF (1.000) > MF+HyDE (0.522) > MQR+MF (0.256) > full composition (0.241).

In other words, the more expansion techniques are stacked on top of MF, the lower its Context P@8 becomes.

Consistent with this, CVR rises as more expansion paths are added: 0% at C3 → 7.5% at C7 → 27.5% at C5 → 37.5% at C8.

3 MQR and HyDE without MF do not improve on the baseline

Compared with C1 (Context P@8 0.181; CVR 37.5%), pure MQR is comparable to slightly lower (0.162; 37.5%) and pure HyDE is worse (0.116; CVR 52.5%).

This is in line with warnings in the literature that query expansion can produce topic drift when the additional candidates are not anchored to the correct constraints, raising apparent recall while simultaneously lowering precision.

Mechanistic Explanation: Why C8 Falls Behind

The cause lies in the composition architecture: in this implementation MF operates as one of the retrieval paths merged through RRF, not as a global filter. Consequently the MQR and HyDE paths contribute candidates that are retrieved without metadata constraints, so the fused result list is once again contaminated by venues that violate the constraints, diluting the pure MF signal.

The Wilcoxon Test Reinforces This Reading

Against C8: C3 and C7 differ significantly with higher scores (p < 0.001), so their advantage is not merely a sampling coincidence. C1 (p = 0.004), C2 (p < 0.001), C4 (p < 0.001), and C6 (p = 0.001) also differ significantly; only C5 does not differ significantly from C8 (p = 0.272). Accordingly, C8 does not occupy the best position and is in fact statistically outperformed by two simpler configurations. Proposition P1 is therefore not supported.

CHAPTER IV · Section 4.2

Constraint Compliance Verification: Why the CVR of C3 = 0%

A concrete illustration: tracing how every criterion of test query Q05 is satisfied by the system's top recommendation under configuration C3.

Query Criterion	Constraint	System Output (Rank 1)	Status
City	Bandung	Grand Serela Bandung (Bandung)	✔ Satisfied
Capacity	≥ 300 pax	350 pax	✔ Satisfied
Package scheme	Fullday	Fullday package available	✔ Satisfied
Budget	≤ IDR 450,000/pax	IDR 420,000/pax	✔ Satisfied

Table IV.3 Constraint Compliance Verification for Test Query Q05 (Configuration C3). Source: the author's experimental results (2026), further processed

Why the Figure of 1.000 Is Not a Miracle

When MF is the only active path, the metadata filter is compiled directly from the query constraints (city, minimum star rating, minimum capacity, budget ceiling), so that only venues passing that filter enter the candidate set.

Yet the ground truth of a query is defined precisely as the venues satisfying all of those criteria. Every retrieved document is therefore necessarily relevant by the ground truth definition, and that is why its CVR is likewise 0%.

This is a logical consequence of the mechanism, not a measurement anomaly.

Performance comparison across ablation configurations — **Figure IV.5** Performance Comparison Across Ablation Configurations: (a) Context Precision@8 and (b) Constraint Violation Rate. *Source: the author's experimental results (2026)*

CHAPTER IV · Section 4.2.1

Cross-Model Robustness Test

Testing whether the finding of Metadata Filtering dominance depends on the particular choice of generator model and evaluation framework.

Changes made: the generator was replaced with GPT-4o; answer quality was assessed using the RAGAS framework with GPT-4.1 as the judge (LLM-as-judge). All retrieval components, pipeline parameters (three MQR variants, 20 candidates per path, 8 context documents, RRF constant 60), and the knowledge base were kept the same. The query set was expanded to 48 items and split into two subgroups: 40 hard queries (containing explicit constraints) and 8 semantic queries (preference-nuanced requests without strict numeric constraints).

Configuration	Hard Queries	Semantic Queries
C1 (baseline)	0.729	0.594
C2 (MQR)	0.595	0.575
C3 (MF)	0.823	0.509
C4 (HyDE)	0.577	0.431
C5 (MQR + MF)	0.707	0.531
C6 (MQR + HyDE)	0.584	0.340
C7 (MF + HyDE)	0.719	0.629
C8 (full)	0.609	0.554

Table IV.4 RAGAS Composite per Query Subgroup in the Cross-Model Robustness Test. Source: the author's robustness test (2026); generator GPT-4o, judge GPT-4.1 (RAGAS)

On Hard Queries: The Pattern Is Replicated

C3 (MF only) obtains the highest score, 0.823, followed by C7 (0.719) and C5 (0.707), while the full composition C8 again does not occupy the top position (0.609). This consistency emerges across a different generator model and a different evaluation framework, strengthening the confidence that the dominance of MF is a property of the constrained retrieval task rather than an artefact of a single model or a single measurement approach.

On Semantic Queries: The Pattern Reverses, and It Is Theoretically Consistent

C7 (MF+HyDE) is in fact the highest (0.629), whereas C3 reaches only 0.509. The explanation: the benefit of query expansion depends on the character of the task. When a request is preference-oriented in nature without strict numerical constraints, there is no constraint that must be safeguarded, so the diversity of candidates produced by HyDE actually becomes advantageous.

For that reason, the conclusion of this study is not that "MF is always best", but rather that "in constrained entity retrieval, MF is more decisive than the stacking of expansion techniques".

It Also Reveals a Limitation of the LLM Judge

In the hard subgroup, the naive baseline C1 obtains a relatively high RAGAS score (0.729) even though the objective metrics show that this configuration is weak (Context P@8 0.181; CVR 37.5%). The reason is that RAGAS assesses the quality and faithfulness of the answer with respect to the context, not the compliance of the recommendation with numerical constraints. An answer that is fluent yet recommends a constraint-violating venue is still rated favourably.

This test is supplementary and is not merged with the main results; its scores are not compared number by number with Table IV.2 because the model, the metrics, and the number of queries differ. Its role is to confirm the direction of the findings, not to replace them.

CHAPTER IV · Section 4.1.3

System Interface and Usage Flow

The Vento AI prototype was successfully implemented as a publicly accessible web chatbot. Users type their event requirements freely in Indonesian, and the system replies with a set of venue recommendations together with a summary of capacity, package schemes, indicative reference prices, and a justification of the fit with the requested criteria. The system also states explicitly that prices are indicative and that final availability needs to be confirmed with the hotel.

Vento AI interface: conversational response and recommendation cards — **Figure IV.2** Vento AI Interface: Conversational Response and Recommendation Cards. *Source: system screenshot by the author (2026)*

Booking request flow through to confirmation sent — **Figure IV.3** Booking Request Flow Through to Confirmation Sent: entry of the event date, number of participants, WhatsApp contact, and budget. *Source: system screenshot by the author (2026)*

CHAPTER IV · Section 4.3 · ANSWERING RQ2

User Evaluation Results

Respondent Profile

Category	Count	Notes
Event organizer owners	12	Target population
EO staff / freelancers	6	Target population
EO vendors / partners	2	Target population
Companies using EO services	1	Target population
Tour guides	1	Target population
Other professions (teachers, civil servants, lawyers, general employees)	9	Outside the target population, analysed separately
Total responses received	31	Experience > 5 years: 26 respondents (83.9%)

Table IV.5 Respondent Profile. Data collection up to 16 July 2026. Source: the author's primary data (2026)

Application of the Inclusion–Exclusion Criteria

Of the 31 responses: 9 were excluded for being outside the target population, 2 were excluded because of blank SUS items, and 2 were excluded for failing the consistency check, so the SUS analysis is based on 18 valid responses.

Important note: the TAM results are reported over all 31 responses whose TAM items were complete; the figure of 18 applies to SUS. The exclusion criteria were established before the analysis.

TAM Results: Proposition P2 Supported

4.39

Perceived Usefulness

4.42

Perceived Ease of Use

4.24

Attitude Towards Use

4.18

Intention to Use

TAM score per construct (scale 1–5)

Dashed line = neutral point 3

All four averages lie well above the neutral point of 3, indicating that the system is perceived as useful and easy to use, with a strong intention to use it again. Since the majority of respondents have more than five years of experience, this result also validates the real need raised in the background of the study.

SUS Results: Proposition P3 Supported by a Narrow Margin

69.9

mean SUS (n = 18 valid)

70.0

median SUS

50%

9 of 18 respondents above the threshold

35–100

range from lowest to highest score

What Does the Figure 68 Mean?

The figure of 68 is neither an industry standard nor a formal pass threshold, but an empirical average compiled by Sauro and Lewis from thousands of SUS questionnaires across hundreds of product studies. A score above 68 means being above the 50th percentile of the products that have been measured with SUS, consistent with the average of 70.1 across 2,324 surveys reported by Bangor et al.

The data support P3, but by a narrow margin, only 1.9 points above the threshold, so adding valid respondents becomes important in order to confirm the stability of the finding.

SUS score vs the normative average

Dashed line = SUS normative threshold of 68

Five Themes from the Open-Ended Feedback

1 UI/UX & first impression

Refinement of the interface, including requests for a more modern appearance.

2 Enrichment of venue information

Venue photos, strengths and weaknesses, and details beyond price. Directly related to retrieval quality.

3 Process indicator

So that users do not assume the application is malfunctioning while the system is working.

4 Consistency & response speed

Directly related to context assembly.

5 Broader coverage

A wider range of cities and venue data sources.

Reading

What holds the usability score back lies in interface aspects and process feedback, not in the substantive quality of the recommendations. Themes 1, 3, and 5 become recommendations for product development.

CHAPTER IV · Section 4.4

Discussion and Design Implications

Regarding RQ1

The main finding: the effectiveness of Advanced RAG in the constrained venue recommendation task is determined primarily by metadata filtering, not by the stacking of query expansion techniques. This result places the study in a position different from the premise 'the more techniques the better' that underlies Proposition P1, and instead offers a more nuanced understanding of when a retrieval technique is beneficial.

A Nuance Regarding RAG-Fusion

Rackauckas reports improvements through multi-query and rank fusion. In open-topic retrieval, enriching the query can indeed increase recall. However, in constrained entity retrieval such as venue selection, additional queries that are not anchored to the constraints instead inject candidates that violate the criteria and lower precision. The benefit of query expansion depends on the character of the task, and in constrained tasks it can become counterproductive if it is not filtered.

Regarding Wijaya & Jayadianti and Karlović

A direct comparison with [16] was not carried out because the metrics, the corpus scale (1 hotel vs 1,741 venues), and the nature of the task (B2C information service vs constrained B2B entity recommendation) differ substantially. What can be concluded qualitatively is that, on a large and multi-constraint corpus, pure semantic similarity is not sufficient: the Context P@8 of the C1 baseline is only 0.181. Karlović's finding [17] that the choice of components strongly affects the results is likewise reinforced, in the form of the sensitivity of the results to the composition of retrieval techniques.

Design Implication: A Proposal for Placing MF

Proposal

Metadata filtering should be applied as a global constraint (a hard pre-filter over all retrieval paths), rather than as one path merged on equal footing through RRF. With such a placement, the recall benefits of MQR and HyDE could potentially be obtained without sacrificing constraint compliance, because all candidates from any path would still be subject to the filter.

Stated honestly: this is still an untested hypothesis, and it is proposed as further research in Chapter V, not claimed as a result.

Comparison of Metadata Filtering placement strategies — **Figure IV.7** Comparison of Metadata Filtering Placement Strategies: MF as a fused path (the one tested) versus MF as a global hard gate (the proposal). The same components, a different placement, opposite semantics. *Source: prepared by the author (2026)*

Regarding RQ2

The high TAM scores across all constructs (4.18–4.42) are in line with the premise of Chapter I that there is a real need among practitioners for a venue information channel that can be accessed independently; an intention to use again of 4.18 from respondents of whom 83.9% have more than five years of experience constitutes a strong signal validating the need.

On the usability side, the SUS score of 69.9 exceeds the threshold of 68 but only narrowly. The open-ended answers show that what holds the score back is not the quality of the recommendations, but rather interface aspects and process feedback, consistent with the finding of Bangor et al. that the SUS score reflects the overall experience, not functional accuracy alone.

The Separation This Study Was Designed Around

Between engine quality (measured by the ablation) and user experience (measured by TAM/SUS): the two are complementary and do not substitute for each other. A system that is accurate but unused is just as useless as a system that is easy to use but wrong.

Limitations Identified at This Stage

The number of valid responses is still below target, so the user findings are provisional.
The narrow SUS margin relative to the normative threshold.
The dataset coverage, although broad (32 provinces), remains a snapshot with indicative prices.

CHAPTER V · Section 5.1

Conclusion Regarding RQ1

An ablation study of eight retrieval technique configurations.

Proposition P1 Not Supported

The full composition of MQR + Metadata Filtering + HyDE with RRF fusion (configuration C8) does not produce the best recommendation quality.

The best configuration is C3, which activates only Metadata Filtering: Context Precision@8, Recommendation Precision@5, Hit Rate, and MRR are all 1.000, with a Constraint Violation Rate of 0%.
Followed by C7 (MF + HyDE) with a Context Precision@8 of 0.522 and a CVR of 7.5%.
The full composition C8 reaches only a Context Precision@8 of 0.241 and a CVR of 37.5%, and is statistically outperformed by both C3 and C7 (Wilcoxon test, p < 0.001).
The technique with the largest contribution is Metadata Filtering: every configuration containing it surpasses every configuration without it.
Adding MQR and HyDE on top of MF actually lowers precision, because both inject candidates that have not been filtered by the constraints into the fused results.

Key Finding for RQ1

In constrained entity retrieval, metadata filtering is more decisive than the stacking of query expansion techniques, and the composition of techniques is not monotonically beneficial.

Why This Finding Sharpens Rather Than Weakens the Study

A study that reports its hypothesis as not supported in fact demonstrates that its testing was genuine and not merely confirmatory. This finding provides a more nuanced understanding of when a retrieval technique is beneficial, something that would not have been obtained had the results simply confirmed the initial assumption.

CHAPTER V · Section 5.1

Conclusion Regarding RQ2

Technology Acceptance: P2 Supported

The TAM evaluation with practitioner respondents from the event ecosystem produced scores that were high and consistent across all constructs: perceived usefulness 4.39; perceived ease of use 4.42; attitude 4.24; and intention to use 4.18 on a scale of 5, all well above the neutral point.

With the majority of respondents having more than five years of experience, this result shows that the system is perceived as useful and easy to use by those who understand the problem of venue selection best, and at the same time validates the real need raised in the background of the study.

Usability: P3 Supported by a Narrow Margin

The SUS measurement on valid respondents produced a mean score of 69.9 (median 70.0), above the SUS normative average threshold of 68.

Analysis of the open-ended answers shows that what holds the usability score back lies in interface aspects and process feedback, not in the substantive quality of the recommendations.

The margin of 1.9 points is narrow, and this is stated openly as a limitation.

Overall

The research objective was achieved: the prototype was successfully built and evaluated through triangulation from both the engine side and the human side. It should be emphasised that achieving the objective is not the same as having all propositions supported: the aim of the research is to test, not to prove correct.

CHAPTER V · Section 5.1

Contributions and Limitations

Research Contributions

Theoretical

Empirical evidence concerning the role and limits of the composition of retrieval refinement techniques within the Advanced RAG framework for constrained entity retrieval tasks in the Indonesian-language B2B MICE domain, in particular the finding that metadata filtering is the primary determinant of quality and that stacking query expansion techniques is not always beneficial. This pattern, as far as the author's search extends, has not been tested in prior literature.

Practical

An independent indicative reference that accelerates the preparation of budget estimates and initial quotations for event organisers.

Limitations: Stated Openly

The knowledge base is a snapshot with indicative reference prices, not real-time availability data.
The user evaluation is based on a number of valid respondents that is still approaching the target, so the final figures may shift.
The margin of the SUS score relative to the normative threshold is narrow.
The system was tested on one language model and one embedding model.
The ablation results are tied to a single composition strategy, namely MF as an RRF-fused path instead of a global pre-filter, so the ranking of configurations may differ under other placement strategies.
All claims are limited to the curated knowledge base corpus and are not generalised to the population of all hotels in Indonesia.

CHAPTER V · Section 5.2

Recommendations

A. Recommendations for System Development: Derived from Respondent Feedback

1 Interface & first impression

A more modern layout for the landing page and the search results display, together with a feature for sorting results by budget.

2 Enrichment of venue information

Photos, a summary of strengths and limitations, and details beyond price, so that recommendations can be acted upon without layered verification.

3 Process indicator

The addition of a loading indicator while the system is working, so that users receive feedback that their request is being processed.

4 Consistency & speed

Improvement through caching of retrieval results and answer streaming.

5 Broader coverage

Cities and venue data sources expanded periodically, accompanied by a scheduled dataset updating mechanism.

B. Recommendations for Further Research

1 Broader objects

Extension to standalone convention centres and non-hotel venues, which have different procurement patterns and package structures.

2 Availability integration

Through data collaboration with property managers, so that the system moves from an indicative reference towards operational information.

3 Testing MF placement, a priority

Testing MF as a global hard pre-filter over all retrieval paths, rather than as a path that is also fused by RRF, a direct follow-up to the RQ1 finding. Subsequently, comparing this composition with semantic re-ranking, knowledge graph approaches, and the Modular and agentic RAG paradigms.

4 Generalisation testing

On different language models and embedding models, given that the literature shows the choice of LLM affects the quality of the results.

5 Cross-session personalisation

Based on the user's preference history, so that the system evolves towards a complete conversational recommender system.

6 Evaluation replication

With a larger respondent pool and UTAUT-based path analysis, and by testing the representativeness of the dataset against national hotel statistics.

CHAPTER V · Closing

Thank You

UNIVERSITAS NUSA MANDIRI Faculty of Information Technology · Informatics Study Program (Master's)

Ready For The Question And Answer Session

In constrained-entity retrieval, metadata filtering is more decisive than the number of techniques stacked on top of one another.

AuthorNuryani Mawar Putri · 14240017

SupervisorProf. Dr. Ir. Dwiza Riana, S.Si, MM, M.Kom, IPU, ASEAN.Eng

Study ProgramInformatics (Master's), FTI Universitas Nusa Mandiri

The Q&A tab contains 130 examiner questions together with their answers and is searchable, press Ctrl+K.

Supporting Tab

Vento AI Demo

A publicly accessible web chatbot prototype on the BuatEvent.id platform, the artifact evaluated in RQ2.

Try It Live In Front Of The Examiners

Type the event requirements freely in Indonesian. The system replies with venue recommendations including capacity, package schemes, indicative reference prices, and a justification of the fit against the requested criteria.

Open Vento AI ↗

Before The Defense: Adjust The Demo Link

The link above points to https://buatevent.id. If the prototype address is different, change the DEMO_URL value on the first line of the assets/app.js file. Also prepare a fallback plan: if the network in the defense room fails, the screenshots in CHAPTER IV and the conversation simulation below are sufficient to demonstrate the flow.

Three Example Queries That Are Safe To Demonstrate

1 Fully constrained query

"Find a hotel venue in Bandung for a full-day meeting, around 300 participants, maximum budget IDR 450,000 per person."

Shows four constraints satisfied at once, the core of Table IV.3.

2 Query with accommodation

"I need a hotel in Bogor for a fullboard residential event, 150 pax, twin sharing, 4 star."

Shows the residential package scheme and star rating filtering.

3 Difficult query

"A venue in a small city with a capacity of 1,000 pax and a very low budget."

Shows the honest behavior of the system when no venue satisfies the requirements, it does not fabricate, but instead states the limitation and suggests adjustments.

Fallback If The Network Fails

Vento AI Conversation Simulation

A reconstruction of the conversation flow following Figure IV.2 in the thesis manuscript, which can be presented without an internet connection.

Vento AIReady to help you plan your event

are there hotels in bandung with meeting venue capacity for 100 pax, budget under 600k?

Of course, Vento will help you look again! ✨ For hotels in Bandung with a capacity of 100 pax and a budget under IDR 600,000 per person for a full day package, there are several interesting options.

Hotel Santika Pasirkaliki

3 star · MICE Hotel · fits 401 pax · Bandung

Meeting halfdayIDR 275,000/pax

Meeting fulldayIDR 350,000/pax

All the details are on the card below. If you like, Vento can help you compare them further.

What To Show During The Demo

Free natural-language query the user does not fill in any filter boxes; the constraints are recognized automatically by the Query Parser and then compiled into metadata filters.
Structured recommendation cards capacity, star rating, and reference prices per package scheme are displayed in full, so they can be used directly to draw up a quotation.
Justification of fit the system explains why the venue matches the criteria, rather than merely displaying a list.
Indicative price statement the system states explicitly that the prices are indicative and that final availability must be confirmed with the hotel.
Follow-up flow from recommendation to booking request: event date, number of participants, WhatsApp contact, budget, through to the confirmation being sent.

Limits To Keep In Mind When Questioned

The system does not cover transactions, real-time availability, or dynamic pricing, all of which remain the domain of the hotel, as stated in the scope (Section 1.4).

Connecting RQ1 And RQ2

From The Ablation Results To The Prototype

RQ1: Ablation Study8 configurations × 40 queries = 320 executions

→

Best compositionMF as the main determinant of constraint compliance

→

Vento AI prototypeimplemented & publicly accessible

→

RQ2: TAM & SUSverification of acceptance & usability by practitioners

What To Emphasize If Questioned

Vento AI is an implementation artifact, not the object of the scientific claim. The knowledge claim of this research rests on the results of the ablation study (RQ1); Vento AI serves as the artifact that makes it possible to test the acceptance of that composition by real practitioners (RQ2).

Why The Prototype Is Still Necessary

Without an artifact that is genuinely usable, RQ2 could not be answered, the respondents used the system directly for real task scenarios before filling in the questionnaire, rather than merely assessing a demonstration. This is what makes the TAM scores of 4.18–4.42 meaningful.

Supporting Tab

Reference Papers

56 references used in the thesis. References marked in green have PDF files that can be read directly, scrolled, and downloaded from this site. The rest can be opened through the publisher's DOI link.

Supporting Tab · Defense Aid

Thesis Defense Q&A

130 questions that the examination committee might ask, together with ready-to-use answers, all drawn from the thesis manuscript. Type a keyword to search, for example "C8", "SUS", "Wilcoxon", "why", or "respondents".

Supporting Tab

About The Researcher

This research grew out of an encounter between a real industry problem and a technological opportunity, not out of a problem found in a journal.

Student ID 14240017 AI Specialization

Nuryani Mawar Putri

MICE Practitioner · Director of PT Inti Jasa Kreatif · Master's Student in Informatics, UNM

More than 14 years of career experience in the Indonesian corporate event organizer industry, leading events ranging from a small scale to thousands of participants in various cities. It was from that day-to-day practice that I repeatedly encountered the problem that forms the background of this thesis: a hotel venue selection process that is still manual, fragmented, and without a price reference that can be accessed independently.

The knowledge base of 1,741 hotel venues that supports this research is not a dataset I downloaded; I compiled it myself from the domain knowledge gained during that practice, together with searches of public sources, covering 32 provinces, 131 cities/regencies, and 91 hotel groups.

Full name	Nuryani Mawar Putri
Student ID	14240017
Study program	Master's in Informatics, specializing in Artificial Intelligence
Faculty	Faculty of Information Technology, Universitas Nusa Mandiri
Supervisor	Prof. Dr. Ir. Dwiza Riana, S.Si, MM, M.Kom, IPU, ASEAN.Eng
Occupation	Director of PT Inti Jasa Kreatif (since August 2015)
Areas of expertise	Event and MICE management, recommender systems, application of LLM and RAG

Why This Background Is Relevant To The Research

Industry experience made it possible for me to construct 40 test queries that reflect real demand patterns rather than artificial queries, and to verify their ground truth against the MICE package structures that genuinely apply in the field (halfday, fullday, fullboard, and residential schemes). The same background also opened access to experienced practitioner respondents: 83.9% of the respondents have more than five years of work experience.

Educational Background

Master's in Informatics, Universitas Nusa Mandiri2024 – present · Artificial Intelligence specialization
Bachelor's in Information Systems, Universitas Nusa MandiriGraduated 2024
Diploma III in Informatics Management, Universitas Bina Sarana InformatikaGraduated 2011

SMA Negeri 11 Jakarta (senior high school)Graduated 2008
SMP Negeri 236 Jakarta (junior high school)Graduated 2005
SD Negeri 07 Pagi Penggilingan, Jakarta (primary school)Graduated 2002

Competency Certifications

1 Program Analyst

LSP Universitas Nusa Mandiri

2 Certified Event Venue Management

LSP MICE, the certification closest to the domain of this research

3 Performance Management Cluster

LSP Jana Dharma

4 Tour Manager Leader

LSP Pariwisata Nusantara

5 Event Marketing Communication

LSP MICE

Employment and Organizational Roles

Director, PT Inti Jasa Kreatif

August 2015 – present. The event organizer company in which this research was carried out, and also the developer of the BuatEvent.id platform on which the Vento AI prototype is implemented.

KEMBERIN (Deputy Chairperson) HIPMI (Division 8) IVENDO (MICE Division) AELI (Public Relations and Publications Division) ASITA (Member)

It is this association membership that opened the way for domain validation as well as for recruiting MICE practitioner respondents in the RQ2 evaluation.

Scientific Publications During The Study Period

No	Title	Published	Relation to the Thesis
1	N. M. Putri and I. Saputra, “A Multimodal Graph-Based Recommendation Architecture for Vendor Discovery in Context-Aware Event Planning”	Jurnal Ilmu dan Teknologi Rekayasa Baru (JNEST), vol. 5, no. 1, 2026	A direct foundation, from graph-based vendor recommendation to RAG-based venue recommendation
2	N. M. Putri and I. Saputra, “A Multimodal Graph-Based Recommendation Architecture for Vendor Discovery in Context-Aware Event Planning”	Jurnal Infotel, vol. 17, no. 4, pp. 825–838, 2025	Experience in building question-answering datasets and a QA–IR pipeline together with the UNM research team
3	N. M. Putri and Y. E. Achyani, “Quality Analysis of the Service Information System at PT. Inti Jasa Kreatif Using the WebQual 4.0 Method”	Jurnal Saintekom, vol. 14, no. 2, 2024	Experience in evaluating web service quality, which enriched the construction of the TAM and SUS questionnaire items

All three publications are listed on the List of Publications Produced page of the thesis manuscript.

The Common Thread

14 years of MICE practiceencountering the venue selection problem first-hand

→

Graph-based vendor recommendationJNEST 2026 publication

→

QA dataset for LLMInfotel 2025 publication

→

This thesisAdvanced RAG for MICE venue recommendation