mirror of
https://github.com/ikawrakow/ik_llama.cpp.git
synced 2026-06-28 04:30:15 -05:00
* Autoparser - complete refactoring of parser architecture Autoparser: add optional argument reshuffle capability Autoparser: True streaming (#20177) * Relax atomicity constraint for nicer, more pleasent, True Streaming parsing * Whitespace * Remove redundant atomics Revert to OAI-compatible args (#20213) * Revert to OAI-compatible args * Apply workaround::func_args_not_string Fix structured outputs (#20223) * Fix structured outputs * Update common/chat-auto-parser-generator.cpp Co-authored-by: Aldehir Rojas <hello@alde.dev> --------- Co-authored-by: Aldehir Rojas <hello@alde.dev> Fix compile bug (#20203) * Fix compile bug * Update common/chat-auto-parser-helpers.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> # Conflicts: # common/chat-auto-parser-helpers.cpp common : gracefully handle incomplete output (#20191) * common : handle incomplete UTF-8 at end of input in PEG parser * cont : if reached end prematurely, emit needs_more_input to propagate partial output * cont: refactor peg parse context to add lenient flag * cont : remove partial flag, keep lenient flag PEG parser for LFM2 (#20251) * PEG parser for LFM2 * Simplify using python_value() common: map developer role to system (#20215) * Map developer role to system * Simplify common: consolidate PEG string parsers (#20263) * common : consolidate PEG string parsers * cont : fix json_string_content() examples : fix empty items in json_schema_to_grammar.py [no ci] (#19968) * Fix logic for retrieving schema items in `json_schema_to_grammar.py` If `schema['items']` is `{}` and `prefixItems not in schema', as `{}` is Falsy, the original code here will raise an error. I think if `schema['items']` is `{}`, them items should just be `{}` * Apply suggestion from @CISC Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Add tests for arrays with empty items Add two unit tests to `tests/test-json-schema-to-grammar.cpp` that validate handling of arrays when 'items' is an empty schema and when 'prefixItems' is present alongside an empty 'items'. Both tests expect the same generated grammar, ensuring the JSON Schema->grammar conversion treats an empty 'items' schema (and the presence of 'prefixItems') correctly and covering this edge case. --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Reduce level of content parser warning message to avoid log spam on non-debug verbosity (#20347) do not return if template parse failed add arg to enable parallel tool call common : fix incorrect uses of stoul (#20313) # Conflicts: # common/arg.cpp # src/llama-grammar.cpp examples : fix empty items in json_schema_to_grammar.py [no ci] (#19968) * Fix logic for retrieving schema items in `json_schema_to_grammar.py` If `schema['items']` is `{}` and `prefixItems not in schema', as `{}` is Falsy, the original code here will raise an error. I think if `schema['items']` is `{}`, them items should just be `{}` * Apply suggestion from @CISC Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Add tests for arrays with empty items Add two unit tests to `tests/test-json-schema-to-grammar.cpp` that validate handling of arrays when 'items' is an empty schema and when 'prefixItems' is present alongside an empty 'items'. Both tests expect the same generated grammar, ensuring the JSON Schema->grammar conversion treats an empty 'items' schema (and the presence of 'prefixItems') correctly and covering this edge case. --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Add support for MiroThinker with new jinja template common/parser: handle reasoning budget (#20297) * v1 * Finished! * Handlie cli * Reasoning sampler * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Less explosive terminology :) * Add utf-8 case and tests * common : migrate reasoning budget sampler to common * cont : clean up * cont : expose state and allow passing as initial state * cont : remove unused imports * cont : update state machine doc string --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Alde Rojas <hello@alde.dev> common/parser: use nlohmann::ordered_json to preserve parameter order (#20385) common/parser: add GigaChatV3/3.1 models support (#19931) Co-authored-by: Mishusha <pmv26021975@gmail.com> common/parser: gracefully handle undetected tool parser, print error message. (#20286) fix: prevent nullptr dereference (#20552) common : fix iterator::end() dereference (#20445) # Conflicts: # common/regex-partial.cpp jinja : add capability check for object args (#20612) common/parser: add `--skip-chat-parsing` to force a pure content parser. (#20289) * Add `--force-pure-content` to force a pure content parser. * Update common/arg.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : rework gpt-oss parser (#20393) * common : rework gpt-oss parser * cont : fix gpt-oss tests * cont : add structured output test * cont : rename final to final_msg common : fix gpt-oss content removal (#20745) common/parser: add proper reasoning tag prefill reading (#20424) * Implement proper prefill extraction * Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp * Update tools/server/server-task.cpp * refactor: move grammars to variant, remove grammar_external, handle exception internally * Make code less C++y Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> chat : handle tool calls with no required args in TAG_WITH_TAGGED format (#20764) * chat : handle tool calls with no required args in TAG_WITH_TAGGED format * Update tests/test-chat.cpp [no ci] Co-authored-by: Aldehir Rojas <hello@alde.dev> --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Aldehir Rojas <hello@alde.dev> common/parser : fix out_of_range crash in throw path (#20424 regression) (#20777) * chat : fix out_of_range crash in throw path (#20424 regression) #20424 introduced effective_input = generation_prompt + input, but the throw path uses input.substr(result.end) where result.end is a position within effective_input. Every thinking model with a non-empty generation_prompt crashes with std::out_of_range instead of the intended error message. Test crashes on unpatched master, passes with fix: cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF cmake --build build --target test-chat ./build/bin/test-chat * Update test-chat.cpp * Update test-chat.cpp * Update test-chat.cpp --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> jinja : fix heap OOB read in value equality comparison (#20782) Address GHSA-q9j6-4hhc-rq9p and GHSA-2q4c-9gq5-5vfp. The three-iterator overload of std::equal in value_array_t::equivalent() and value_object_t::equivalent() reads past the end of the shorter container when comparing arrays or objects of different lengths. Use the four-iterator overload (C++14) which checks both range lengths. Found-by: Pwno common : fix typo in debug log ('extracft' -> 'extract') (#20807) common/parser: fix nasty bug causing subtle corruption of generation prompt (#20825) jinja : refactor token advancement (#20864) * refactor token advancement * exercise sub-expressions common/autoparser : detect reasoning markers when enable_thinking changes system prompt (#20859) common : replace wrap_for_generation with a prefix convenience function and fix gpt-oss (#20912) jinja: fix macro with kwargs (#20960) * jinja: fix macro with kwargs * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix newline problem --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : inhibit lazy grammar sampler while reasoning is active (#20970) * common : inhibit grammar while reasoning budget is active * cont : update force_pos in accept * cont : fix tests * cont : tweak should apply logic * cont : return early not using grammar sampler * Add tests * cont : prevent backend sampling when reasoning budget enabled * cont : fix typo --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> # Conflicts: # common/reasoning-budget.h # common/sampling.cpp # tools/cli/cli.cpp # tools/server/server-common.cpp # tools/server/server-task.cpp common/parser: fix reasoning whitespace bugs + extra parser tests (#21085) * fix whitespace reasoning issues + add reconstruction tests * Proper fix * fix Nemotron autoparser test expectations to include newline in marker common : add reasoning_format = none support to gpt-oss (#21094) common/json-schema: fix: handle non-capturing groups (?:...) in JSON schema pattern converter (#21124) The regex-to-grammar converter in _visit_pattern() crashes with SIGSEGV when a JSON schema "pattern" field contains a non-capturing group (?:...). Root cause: when the parser sees '(' followed by '?', it pushes a warning but does not advance past '?:'. The recursive transform() call then interprets '?' as a quantifier and calls seq.back() on an empty vector, causing undefined behavior. This commonly occurs when serving OpenAI-compatible tool calls from clients that include complex regex patterns in their JSON schemas (e.g., date validation patterns like ^(?:(?:\d\d[2468][048]|...)-02-29|...)$). The fix: - Skip '?:' after '(' to treat non-capturing groups as regular groups - For unsupported syntax (?=, ?!, etc.), skip to matching ')' safely, handling escaped characters to avoid miscounting parenthesis depth - Adjust the ')' unbalanced-parentheses check using direct char comparisons instead of substr - Add test cases for non-capturing groups (C++ only, as the JS/Python implementations do not yet support this syntax) common/parser: fix handling of tool definition with missing properties key (#21128) jinja : handle empty expressions correctly (#20913) * Reject empty computed member expressions before returning slices[0] from parse_member_expression_arguments(). * Treat empty computed member expressions with Jinja2 undefined semantics Treat empty computed member expressions like `a[]` as undefined instead of raising a parser error, to match Jinja2 behavior. - return a noop expression for empty computed member arguments - return undefined when a computed member key evaluates to undefined - add Jinja tests covering `a[]|default('fallback')` and `a[] is undefined` * Handle undefined computed member properties Move undefined-property handling to the common member access path, and add a test covering `a[undefined] is undefined`. * Use default undefined value in member access Initialize val and then return it when property is undefined. Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * empty statement parses to blank_expression instead of noop_statement --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : gpt-oss handle builtin and unsolicited tool calls (#21213) fix: tool call parsing for LFM2 and LFM2.5 models (#21242) * fix: tool call parsing for LFM2 and LFM2.5 models' * refactor: add test / break out lfm2 and lfm2.5 parsing logic # Conflicts: # common/chat.cpp Relax prefill parser to allow space. (#21240) * Relax prefill parser to allow space. * Move changes from prefix() to parser generation * Only allow spaces if we're not having a pure content parser next common : add commentary rules for gpt-oss-20b (#21286) add reasoning budget model, mtmd: fix gguf conversion for audio/vision mmproj (#21309) * fix gguf conversion for audio/vision mmproj * fix test # Conflicts: # convert_hf_to_gguf.py # examples/eval-callback/eval-callback.cpp # examples/mtmd/CMakeLists.txt # examples/mtmd/clip-impl.h # examples/mtmd/mtmd.cpp # gguf-py/gguf/constants.py # gguf-py/gguf/gguf_writer.py # gguf-py/gguf/tensor_mapping.py # src/CMakeLists.txt # src/llama-arch.cpp # src/llama-arch.h # src/llama-model.cpp # src/llama-model.h # src/llama-vocab.cpp # src/models/models.h # tests/test-llama-archs.cpp # tools/mtmd/clip-graph.h # tools/mtmd/clip-model.h # tools/mtmd/clip.cpp # tools/mtmd/models/models.h fix: gemma 4 template (#21326) chat : avoid including json in chat.h (#21306) jinja: coerce input for string-specific filters (#21370) common : fix tool call type detection for nullable and enum schemas (#21327) * common : fix tool call type detection for nullable and enum schemas * common, tests : fix grammar delegation for nullable/enum schemas and add tests Fix enum type inference to scan all enum values (not just index 0) so schemas like {"enum": [0, "celsius"]} correctly detect string type. Fix schema_delegates in peg-parser to handle nullable type arrays (["string", "null"]) and typeless enum schemas in raw mode, allowing the tagged parser to use raw text instead of JSON-formatted strings. Add test cases for Qwen3-Coder (TAG_WITH_TAGGED format): - nullable string ["string", "null"] - nullable string with null first ["null", "string"] - nullable integer ["integer", "null"] - enum without explicit type key common/parser: fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers (#21230) * Fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers * Rename * Update common/chat-auto-parser-generator.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : add gemma 4 specialized parser (#21418) * common : add gemma4 dedicated parser * cont : add '<|tool_response>' as eog * cont : emit JSON from Gemma4 tool call AST * cont : more fixes * cont : refactor convert function * cont : refine rules and mapping * cont : add more tests * cont : clean up * cont : remove autoparser gemma4 implementation * cont : more cleanup * cont : rename gemma4.jinja to match the others * cont : add custom template to support interleaved thinking * cont : preserve reasoning in model turns * cont : fix initializer error * cont : fix unused vars * cont : fix accidental static * cont : fix specialized_template signature * fix extra semicolon * remove debug line and extra space [no ci] fix reasoning budget parser: fix MiniMax handling (#21573) jinja : support ensure_ascii=true, string repetition and int/float self-filtering (#21623) * feat: jinja engine improvements for reka-edge Port three Jinja engine improvements needed for the reka-edge model: 1. Python-style string repetition ("ab" * 3 → "ababab") 2. ensure_ascii=true support for tojson filter (escapes non-ASCII to \uXXXX) 3. int() builtin on value_int_t (identity, needed for Reka Edge template) * fix: escape invalid utf8 bytes when ensure_ascii=true The json_ensure_ascii_preserving_format function does not correctly handle an edge case where if UTF-8 parsing fails, it adds the non-ascii character back to the output as a raw byte. This commit fixes that by adding the unicode standard replacement character \\ufffd to the output instead. This is the standard behavior for various programming languages like Python, Rust, Go, etc. * chore: address PR comments 1. Add todo comment for supporting string repetition for array/tuples 2. Add support for float identity operation 3. Move invalid ascii test case to test_fuzzing * chore: accept suggestion for common/jinja/value.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : simplify autoparser tagged parser rules (#21216) * common : simplify autoparser tagged parser rules * cont : remove upper limit on optional args * cont : revert changes to parsing at the end * cont : undo arbitrary ordering of optional args * cont : fix uninitialized required parameters * revert to simplify merge * re-apply patches * restore flexible optional arg ordering tests common : fix ambiguous grammar rule in gemma4 (#21661) * common : fix ambiguous grammar rule in gemma4 * cont : fix missing comma... common : enable reasoning budget sampler for gemma4 (#21697) * fix: enable reasoning budget sampler for gemma4 Add thinking_start_tag and thinking_end_tag to common_chat_params_init_gemma4(). Without these, the reasoning budget sampler never activates for gemma4. Make the newline after "thought" optional in the PEG parser to handle budget=0 (sampler forces end tag before the newline). Add test case for empty thinking block. Fixes #21487 * use p.space() instead of p.optional(p.literal("\n")) in gemma4 thought parser common : better align to the updated official gemma4 template (#21704) fix: Fix broken structured output when using $refs in json_schema (#21699) chat: dedicated DeepSeek v3.2 parser + "official" template (#21785) Hide render_message_to_json warning common/gemma4 : handle parsing edge cases (#21760) common: skip reasoning budget sampler when no budget is requested (#21870) * common: skip reasoning budget sampler when no budget is requested After I added thinking_start_tag / thinking_end_tag for gemma4 in #21697, the reasoning budget sampler gets unconditionally created even when no budget is configured (the default -1). The same applies to kimi_k2, lfm2, lfm2_5, and ministral_3 which also set these tags. The budget gets converted to INT_MAX, so the sampler never actually forces any tokens but still runs per-token checks (start tag matching in IDLE state, token-to-piece conversion + UTF-8 checks in COUNTING state). More importantly, the mere existence of the sampler (non-null rbudget) disables backend sampling. Backend sampling lets the GPU select tokens directly, avoiding a full logits transfer from GPU to CPU every token. This could explain the 30% speed regression reported in #21784 (98 t/s to 70 t/s on Vulkan). So I added a reasoning_budget_tokens >= 0 check to the sampler creation condition. When the budget is unlimited, the sampler is not created, backend sampling stays enabled, and no per-token overhead is added. When a budget is explicitly set (0, 128, 1024, etc.), the sampler is created and works as before. * common: preserve rbudget when grammar is lazy Following up on the review feedback on #21870: keep the reasoning budget sampler when grammar_lazy is true, so the thinking-block grammar suppression from #20970 still works when tools are in use. This way, we only skip the sampler when both no budget is set AND grammar is not lazy. autoparser: support case of JSON_NATIVE with per-call markers (test case: Reka-Edge) (#21892) * fix grammar * fix add sampled token --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: firecoperana <firecoperana>
907 lines
36 KiB
C++
907 lines
36 KiB
C++
#include "lexer.h"
|
|
#include "runtime.h"
|
|
#include "value.h"
|
|
#include "utils.h"
|
|
|
|
#include <string>
|
|
#include <vector>
|
|
#include <memory>
|
|
#include <cmath>
|
|
|
|
#define FILENAME "jinja-runtime"
|
|
|
|
bool g_jinja_debug = false;
|
|
|
|
namespace jinja {
|
|
|
|
void enable_debug(bool enable) {
|
|
g_jinja_debug = enable;
|
|
}
|
|
|
|
static value_string exec_statements(const statements & stmts, context & ctx) {
|
|
auto result = mk_val<value_array>();
|
|
for (const auto & stmt : stmts) {
|
|
JJ_DEBUG("Executing statement of type %s", stmt->type().c_str());
|
|
result->push_back(stmt->execute(ctx));
|
|
}
|
|
// convert to string parts
|
|
value_string str = mk_val<value_string>();
|
|
gather_string_parts_recursive(result, str);
|
|
return str;
|
|
}
|
|
|
|
static std::string get_line_col(const std::string & source, size_t pos) {
|
|
size_t line = 1;
|
|
size_t col = 1;
|
|
for (size_t i = 0; i < pos && i < source.size(); i++) {
|
|
if (source[i] == '\n') {
|
|
line++;
|
|
col = 1;
|
|
} else {
|
|
col++;
|
|
}
|
|
}
|
|
return "line " + std::to_string(line) + ", column " + std::to_string(col);
|
|
}
|
|
|
|
static void ensure_key_type_allowed(const value & val) {
|
|
if (!val->is_hashable()) {
|
|
throw std::runtime_error("Type: " + val->type() + " is not allowed as object key");
|
|
}
|
|
}
|
|
|
|
// execute with error handling
|
|
value statement::execute(context & ctx) {
|
|
try {
|
|
return execute_impl(ctx);
|
|
} catch (const continue_statement::signal & /* ex */) {
|
|
throw;
|
|
} catch (const break_statement::signal & /* ex */) {
|
|
throw;
|
|
} catch (const rethrown_exception & /* ex */) {
|
|
throw;
|
|
} catch (const not_implemented_exception & /* ex */) {
|
|
throw;
|
|
} catch (const std::exception & e) {
|
|
const std::string & source = *ctx.src;
|
|
if (source.empty()) {
|
|
std::ostringstream oss;
|
|
oss << "\nError executing " << type() << " at position " << pos << ": " << e.what();
|
|
throw rethrown_exception(oss.str());
|
|
} else {
|
|
std::ostringstream oss;
|
|
oss << "\n------------\n";
|
|
oss << "While executing " << type() << " at " << get_line_col(source, pos) << " in source:\n";
|
|
oss << peak_source(source, pos) << "\n";
|
|
oss << "Error: " << e.what();
|
|
// throw as another exception to avoid repeated formatting
|
|
throw rethrown_exception(oss.str());
|
|
}
|
|
}
|
|
}
|
|
|
|
value identifier::execute_impl(context & ctx) {
|
|
auto it = ctx.get_val(val);
|
|
auto builtins = global_builtins();
|
|
if (!it->is_undefined()) {
|
|
if (ctx.is_get_stats) {
|
|
value_t::stats_t::mark_used(it);
|
|
}
|
|
JJ_DEBUG("Identifier '%s' found, type = %s", val.c_str(), it->type().c_str());
|
|
return it;
|
|
} else if (builtins.find(val) != builtins.end()) {
|
|
JJ_DEBUG("Identifier '%s' found in builtins", val.c_str());
|
|
return mk_val<value_func>(val, builtins.at(val));
|
|
} else {
|
|
JJ_DEBUG("Identifier '%s' not found, returning undefined", val.c_str());
|
|
return mk_val<value_undefined>(val);
|
|
}
|
|
}
|
|
|
|
value object_literal::execute_impl(context & ctx) {
|
|
auto obj = mk_val<value_object>();
|
|
for (const auto & pair : val) {
|
|
value key = pair.first->execute(ctx);
|
|
value val = pair.second->execute(ctx);
|
|
JJ_DEBUG("Object literal: setting key '%s' with value type %s", key->as_string().str().c_str(), val->type().c_str());
|
|
obj->insert(key, val);
|
|
}
|
|
return obj;
|
|
}
|
|
|
|
value binary_expression::execute_impl(context & ctx) {
|
|
value left_val = left->execute(ctx);
|
|
|
|
// Logical operators
|
|
if (op.value == "and") {
|
|
JJ_DEBUG("Executing logical test: %s AND %s", left->type().c_str(), right->type().c_str());
|
|
return left_val->as_bool() ? right->execute(ctx) : std::move(left_val);
|
|
} else if (op.value == "or") {
|
|
JJ_DEBUG("Executing logical test: %s OR %s", left->type().c_str(), right->type().c_str());
|
|
return left_val->as_bool() ? std::move(left_val) : right->execute(ctx);
|
|
}
|
|
|
|
// Equality operators
|
|
value right_val = right->execute(ctx);
|
|
JJ_DEBUG("Executing binary expression %s '%s' %s", left_val->type().c_str(), op.value.c_str(), right_val->type().c_str());
|
|
if (op.value == "==") {
|
|
return mk_val<value_bool>(*left_val == *right_val);
|
|
} else if (op.value == "!=") {
|
|
return mk_val<value_bool>(!(*left_val == *right_val));
|
|
}
|
|
|
|
auto workaround_concat_null_with_str = [&](value & res) -> bool {
|
|
bool is_left_null = left_val->is_none() || left_val->is_undefined();
|
|
bool is_right_null = right_val->is_none() || right_val->is_undefined();
|
|
bool is_left_str = is_val<value_string>(left_val);
|
|
bool is_right_str = is_val<value_string>(right_val);
|
|
if ((is_left_null && is_right_str) || (is_right_null && is_left_str)) {
|
|
JJ_DEBUG("%s", "Workaround: treating null/undefined as empty string for string concatenation");
|
|
string left_str = is_left_null ? string() : left_val->as_string();
|
|
string right_str = is_right_null ? string() : right_val->as_string();
|
|
auto output = left_str.append(right_str);
|
|
res = mk_val<value_string>(std::move(output));
|
|
return true;
|
|
}
|
|
return false;
|
|
};
|
|
|
|
auto test_is_in = [&]() -> bool {
|
|
func_args args(ctx);
|
|
args.push_back(left_val);
|
|
args.push_back(right_val);
|
|
return global_builtins().at("test_is_in")(args)->as_bool();
|
|
};
|
|
|
|
// Handle undefined and null values
|
|
if (is_val<value_undefined>(left_val) || is_val<value_undefined>(right_val)) {
|
|
if (is_val<value_undefined>(right_val) && (op.value == "in" || op.value == "not in")) {
|
|
// Special case: `anything in undefined` is `false` and `anything not in undefined` is `true`
|
|
return mk_val<value_bool>(op.value == "not in");
|
|
}
|
|
if (op.value == "+" || op.value == "~") {
|
|
value res = mk_val<value_undefined>();
|
|
if (workaround_concat_null_with_str(res)) {
|
|
return res;
|
|
}
|
|
}
|
|
throw std::runtime_error("Cannot perform operation " + op.value + " on undefined values");
|
|
} else if (is_val<value_none>(left_val) || is_val<value_none>(right_val)) {
|
|
if (op.value == "+" || op.value == "~") {
|
|
value res = mk_val<value_undefined>();
|
|
if (workaround_concat_null_with_str(res)) {
|
|
return res;
|
|
}
|
|
}
|
|
throw std::runtime_error("Cannot perform operation on null values");
|
|
}
|
|
|
|
// Float operations
|
|
if ((is_val<value_int>(left_val) || is_val<value_float>(left_val)) &&
|
|
(is_val<value_int>(right_val) || is_val<value_float>(right_val))) {
|
|
double a = left_val->as_float();
|
|
double b = right_val->as_float();
|
|
if (op.value == "+" || op.value == "-" || op.value == "*") {
|
|
double res = (op.value == "+") ? a + b : (op.value == "-") ? a - b : a * b;
|
|
JJ_DEBUG("Arithmetic operation: %f %s %f = %f", a, op.value.c_str(), b, res);
|
|
bool is_float = is_val<value_float>(left_val) || is_val<value_float>(right_val);
|
|
if (is_float) {
|
|
return mk_val<value_float>(res);
|
|
} else {
|
|
return mk_val<value_int>(static_cast<int64_t>(res));
|
|
}
|
|
} else if (op.value == "/") {
|
|
JJ_DEBUG("Division operation: %f / %f", a, b);
|
|
return mk_val<value_float>(a / b);
|
|
} else if (op.value == "%") {
|
|
double rem = std::fmod(a, b);
|
|
JJ_DEBUG("Modulo operation: %f %% %f = %f", a, b, rem);
|
|
bool is_float = is_val<value_float>(left_val) || is_val<value_float>(right_val);
|
|
if (is_float) {
|
|
return mk_val<value_float>(rem);
|
|
} else {
|
|
return mk_val<value_int>(static_cast<int64_t>(rem));
|
|
}
|
|
} else if (op.value == "<") {
|
|
JJ_DEBUG("Comparison operation: %f < %f is %d", a, b, a < b);
|
|
return mk_val<value_bool>(a < b);
|
|
} else if (op.value == ">") {
|
|
JJ_DEBUG("Comparison operation: %f > %f is %d", a, b, a > b);
|
|
return mk_val<value_bool>(a > b);
|
|
} else if (op.value == ">=") {
|
|
JJ_DEBUG("Comparison operation: %f >= %f is %d", a, b, a >= b);
|
|
return mk_val<value_bool>(a >= b);
|
|
} else if (op.value == "<=") {
|
|
JJ_DEBUG("Comparison operation: %f <= %f is %d", a, b, a <= b);
|
|
return mk_val<value_bool>(a <= b);
|
|
}
|
|
}
|
|
|
|
// Array operations
|
|
if (is_val<value_array>(left_val) && is_val<value_array>(right_val)) {
|
|
if (op.value == "+") {
|
|
auto & left_arr = left_val->as_array();
|
|
auto & right_arr = right_val->as_array();
|
|
auto result = mk_val<value_array>();
|
|
for (const auto & item : left_arr) {
|
|
result->push_back(item);
|
|
}
|
|
for (const auto & item : right_arr) {
|
|
result->push_back(item);
|
|
}
|
|
return result;
|
|
}
|
|
} else if (is_val<value_array>(right_val)) {
|
|
// case: 1 in [0, 1, 2]
|
|
bool member = test_is_in();
|
|
if (op.value == "in") {
|
|
return mk_val<value_bool>(member);
|
|
} else if (op.value == "not in") {
|
|
return mk_val<value_bool>(!member);
|
|
}
|
|
}
|
|
|
|
// String concatenation with ~ and +
|
|
if ((is_val<value_string>(left_val) || is_val<value_string>(right_val)) &&
|
|
(op.value == "~" || op.value == "+")) {
|
|
JJ_DEBUG("String concatenation with %s operator", op.value.c_str());
|
|
auto output = left_val->as_string().append(right_val->as_string());
|
|
auto res = mk_val<value_string>();
|
|
res->val_str = std::move(output);
|
|
return res;
|
|
}
|
|
|
|
// Python-style string repetition
|
|
// TODO: support array/tuple repetition (e.g., [1, 2] * 3 → [1, 2, 1, 2, 1, 2])
|
|
if (op.value == "*" &&
|
|
((is_val<value_string>(left_val) && is_val<value_int>(right_val)) ||
|
|
(is_val<value_int>(left_val) && is_val<value_string>(right_val)))) {
|
|
const auto & str = is_val<value_string>(left_val) ? left_val->as_string() : right_val->as_string();
|
|
const int64_t repeat = is_val<value_int>(right_val) ? right_val->as_int() : left_val->as_int();
|
|
auto res = mk_val<value_string>();
|
|
if (repeat <= 0) {
|
|
return res;
|
|
}
|
|
for (int64_t i = 0; i < repeat; ++i) {
|
|
res->val_str = res->val_str.append(str);
|
|
}
|
|
return res;
|
|
}
|
|
|
|
// String membership
|
|
if (is_val<value_string>(left_val) && is_val<value_string>(right_val)) {
|
|
// case: "a" in "abc"
|
|
bool member = test_is_in();
|
|
if (op.value == "in") {
|
|
return mk_val<value_bool>(member);
|
|
} else if (op.value == "not in") {
|
|
return mk_val<value_bool>(!member);
|
|
}
|
|
}
|
|
|
|
// Value key in object
|
|
if (is_val<value_object>(right_val)) {
|
|
// case: key in {key: value}
|
|
bool member = test_is_in();
|
|
if (op.value == "in") {
|
|
return mk_val<value_bool>(member);
|
|
} else if (op.value == "not in") {
|
|
return mk_val<value_bool>(!member);
|
|
}
|
|
}
|
|
|
|
throw std::runtime_error("Unknown operator \"" + op.value + "\" between " + left_val->type() + " and " + right_val->type());
|
|
}
|
|
|
|
static value try_builtin_func(context & ctx, const std::string & name, value & input, bool undef_on_missing = false) {
|
|
JJ_DEBUG("Trying built-in function '%s' for type %s", name.c_str(), input->type().c_str());
|
|
if (ctx.is_get_stats) {
|
|
value_t::stats_t::mark_used(input);
|
|
input->stats.ops.insert(name);
|
|
}
|
|
auto builtins = input->get_builtins();
|
|
auto it = builtins.find(name);
|
|
if (it != builtins.end()) {
|
|
JJ_DEBUG("Binding built-in '%s'", name.c_str());
|
|
return mk_val<value_func>(name, it->second, input);
|
|
}
|
|
if (undef_on_missing) {
|
|
return mk_val<value_undefined>(name);
|
|
}
|
|
throw std::runtime_error("Unknown (built-in) filter '" + name + "' for type " + input->type());
|
|
}
|
|
|
|
value filter_expression::execute_impl(context & ctx) {
|
|
value input = operand ? operand->execute(ctx) : val;
|
|
|
|
JJ_DEBUG("Applying filter to %s", input->type().c_str());
|
|
|
|
if (is_stmt<identifier>(filter)) {
|
|
auto filter_id = cast_stmt<identifier>(filter)->val;
|
|
|
|
if (filter_id == "trim") {
|
|
filter_id = "strip"; // alias
|
|
}
|
|
JJ_DEBUG("Applying filter '%s' to %s", filter_id.c_str(), input->type().c_str());
|
|
// TODO: Refactor filters so this coercion can be done automatically
|
|
if (!input->is_undefined() && !is_val<value_string>(input) && (
|
|
filter_id == "capitalize" ||
|
|
filter_id == "lower" ||
|
|
filter_id == "replace" ||
|
|
filter_id == "strip" ||
|
|
filter_id == "title" ||
|
|
filter_id == "upper" ||
|
|
filter_id == "wordcount"
|
|
)) {
|
|
JJ_DEBUG("Coercing %s to String for '%s' filter", input->type().c_str(), filter_id.c_str());
|
|
input = mk_val<value_string>(input->as_string());
|
|
}
|
|
return try_builtin_func(ctx, filter_id, input)->invoke(func_args(ctx));
|
|
|
|
} else if (is_stmt<call_expression>(filter)) {
|
|
auto call = cast_stmt<call_expression>(filter);
|
|
if (!is_stmt<identifier>(call->callee)) {
|
|
throw std::runtime_error("Filter callee must be an identifier");
|
|
}
|
|
auto filter_id = cast_stmt<identifier>(call->callee)->val;
|
|
|
|
if (filter_id == "trim") {
|
|
filter_id = "strip"; // alias
|
|
}
|
|
JJ_DEBUG("Applying filter '%s' with arguments to %s", filter_id.c_str(), input->type().c_str());
|
|
func_args args(ctx);
|
|
for (const auto & arg_expr : call->args) {
|
|
args.push_back(arg_expr->execute(ctx));
|
|
}
|
|
|
|
return try_builtin_func(ctx, filter_id, input)->invoke(args);
|
|
|
|
} else {
|
|
throw std::runtime_error("Invalid filter expression");
|
|
}
|
|
}
|
|
|
|
value filter_statement::execute_impl(context & ctx) {
|
|
// eval body as string, then apply filter
|
|
auto body_val = exec_statements(body, ctx);
|
|
value_string parts = mk_val<value_string>();
|
|
gather_string_parts_recursive(body_val, parts);
|
|
|
|
JJ_DEBUG("FilterStatement: applying filter to body string of length %zu", parts->val_str.length());
|
|
filter_expression filter_expr(std::move(parts), std::move(filter));
|
|
value out = filter_expr.execute(ctx);
|
|
|
|
// this node can be reused later, make sure filter is preserved
|
|
this->filter = std::move(filter_expr.filter);
|
|
return out;
|
|
}
|
|
|
|
value test_expression::execute_impl(context & ctx) {
|
|
// NOTE: "value is something" translates to function call "test_is_something(value)"
|
|
const auto & builtins = global_builtins();
|
|
|
|
std::string test_id;
|
|
value input = operand->execute(ctx);
|
|
|
|
func_args args(ctx);
|
|
args.push_back(input);
|
|
|
|
if (is_stmt<identifier>(test)) {
|
|
test_id = cast_stmt<identifier>(test)->val;
|
|
} else if (is_stmt<call_expression>(test)) {
|
|
auto call = cast_stmt<call_expression>(test);
|
|
if (!is_stmt<identifier>(call->callee)) {
|
|
throw std::runtime_error("Test callee must be an identifier");
|
|
}
|
|
test_id = cast_stmt<identifier>(call->callee)->val;
|
|
|
|
JJ_DEBUG("Applying test '%s' with arguments to %s", test_id.c_str(), input->type().c_str());
|
|
for (const auto & arg_expr : call->args) {
|
|
args.push_back(arg_expr->execute(ctx));
|
|
}
|
|
|
|
} else {
|
|
throw std::runtime_error("Invalid test expression");
|
|
}
|
|
|
|
auto it = builtins.find("test_is_" + test_id);
|
|
JJ_DEBUG("Test expression %s '%s' %s (using function 'test_is_%s')", operand->type().c_str(), test_id.c_str(), negate ? "(negate)" : "", test_id.c_str());
|
|
if (it == builtins.end()) {
|
|
throw std::runtime_error("Unknown test '" + test_id + "'");
|
|
}
|
|
|
|
auto res = it->second(args);
|
|
|
|
if (negate) {
|
|
return mk_val<value_bool>(!res->as_bool());
|
|
} else {
|
|
return res;
|
|
}
|
|
}
|
|
|
|
value unary_expression::execute_impl(context & ctx) {
|
|
value operand_val = argument->execute(ctx);
|
|
JJ_DEBUG("Executing unary expression with operator '%s'", op.value.c_str());
|
|
|
|
if (op.value == "not") {
|
|
return mk_val<value_bool>(!operand_val->as_bool());
|
|
} else if (op.value == "-") {
|
|
if (is_val<value_int>(operand_val)) {
|
|
return mk_val<value_int>(-operand_val->as_int());
|
|
} else if (is_val<value_float>(operand_val)) {
|
|
return mk_val<value_float>(-operand_val->as_float());
|
|
} else {
|
|
throw std::runtime_error("Unary - operator requires numeric operand");
|
|
}
|
|
}
|
|
|
|
throw std::runtime_error("Unknown unary operator '" + op.value + "'");
|
|
}
|
|
|
|
value if_statement::execute_impl(context & ctx) {
|
|
value test_val = test->execute(ctx);
|
|
|
|
auto out = mk_val<value_array>();
|
|
if (test_val->as_bool()) {
|
|
for (auto & stmt : body) {
|
|
JJ_DEBUG("IF --> Executing THEN body, current block: %s", stmt->type().c_str());
|
|
out->push_back(stmt->execute(ctx));
|
|
}
|
|
} else {
|
|
for (auto & stmt : alternate) {
|
|
JJ_DEBUG("IF --> Executing ELSE body, current block: %s", stmt->type().c_str());
|
|
out->push_back(stmt->execute(ctx));
|
|
}
|
|
}
|
|
// convert to string parts
|
|
value_string str = mk_val<value_string>();
|
|
gather_string_parts_recursive(out, str);
|
|
return str;
|
|
}
|
|
|
|
value for_statement::execute_impl(context & ctx) {
|
|
context scope(ctx); // new scope for loop variables
|
|
|
|
jinja::select_expression * select_expr = cast_stmt<select_expression>(iterable);
|
|
statement_ptr test_expr_nullptr;
|
|
|
|
statement_ptr & iter_expr = [&]() -> statement_ptr & {
|
|
auto tmp = cast_stmt<select_expression>(iterable);
|
|
return tmp ? tmp->lhs : iterable;
|
|
}();
|
|
statement_ptr & test_expr = [&]() -> statement_ptr & {
|
|
auto tmp = cast_stmt<select_expression>(iterable);
|
|
return tmp ? tmp->test : test_expr_nullptr;
|
|
}();
|
|
|
|
JJ_DEBUG("Executing for statement, iterable type: %s", iter_expr->type().c_str());
|
|
|
|
value iterable_val = iter_expr->execute(scope);
|
|
|
|
// mark the variable being iterated as used for stats
|
|
if (ctx.is_get_stats) {
|
|
value_t::stats_t::mark_used(iterable_val);
|
|
iterable_val->stats.ops.insert("array_access");
|
|
}
|
|
|
|
if (iterable_val->is_undefined()) {
|
|
JJ_DEBUG("%s", "For loop iterable is undefined, skipping loop");
|
|
iterable_val = mk_val<value_array>();
|
|
}
|
|
|
|
if (!is_val<value_array>(iterable_val) && !is_val<value_object>(iterable_val)) {
|
|
throw std::runtime_error("Expected iterable or object type in for loop: got " + iterable_val->type());
|
|
}
|
|
|
|
std::vector<value> items;
|
|
if (is_val<value_object>(iterable_val)) {
|
|
JJ_DEBUG("%s", "For loop over object keys");
|
|
auto & obj = iterable_val->as_ordered_object();
|
|
for (auto & p : obj) {
|
|
auto tuple = mk_val<value_tuple>(p);
|
|
items.push_back(std::move(tuple));
|
|
}
|
|
if (ctx.is_get_stats) {
|
|
value_t::stats_t::mark_used(iterable_val);
|
|
iterable_val->stats.ops.insert("object_access");
|
|
}
|
|
} else {
|
|
JJ_DEBUG("%s", "For loop over array items");
|
|
auto & arr = iterable_val->as_array();
|
|
for (const auto & item : arr) {
|
|
items.push_back(item);
|
|
}
|
|
if (ctx.is_get_stats) {
|
|
value_t::stats_t::mark_used(iterable_val);
|
|
iterable_val->stats.ops.insert("array_access");
|
|
}
|
|
}
|
|
|
|
std::vector<std::function<void(context &)>> scope_update_fns;
|
|
|
|
std::vector<value> filtered_items;
|
|
for (size_t i = 0; i < items.size(); ++i) {
|
|
context loop_scope(scope);
|
|
|
|
value current = items[i];
|
|
|
|
std::function<void(context&)> scope_update_fn = [](context &) { /* no-op */};
|
|
if (is_stmt<identifier>(loopvar)) {
|
|
auto id = cast_stmt<identifier>(loopvar)->val;
|
|
|
|
if (is_val<value_object>(iterable_val)) {
|
|
// case example: {% for key in dict %}
|
|
current = items[i]->as_array()[0];
|
|
scope_update_fn = [id, &items, i](context & ctx) {
|
|
ctx.set_val(id, items[i]->as_array()[0]);
|
|
};
|
|
} else {
|
|
// case example: {% for item in list %}
|
|
scope_update_fn = [id, &items, i](context & ctx) {
|
|
ctx.set_val(id, items[i]);
|
|
};
|
|
}
|
|
|
|
} else if (is_stmt<tuple_literal>(loopvar)) {
|
|
// case example: {% for key, value in dict %}
|
|
auto tuple = cast_stmt<tuple_literal>(loopvar);
|
|
if (!is_val<value_array>(current)) {
|
|
throw std::runtime_error("Cannot unpack non-iterable type: " + current->type());
|
|
}
|
|
auto & c_arr = current->as_array();
|
|
if (tuple->val.size() != c_arr.size()) {
|
|
throw std::runtime_error(std::string("Too ") + (tuple->val.size() > c_arr.size() ? "few" : "many") + " items to unpack");
|
|
}
|
|
scope_update_fn = [tuple, &items, i](context & ctx) {
|
|
auto & c_arr = items[i]->as_array();
|
|
for (size_t j = 0; j < tuple->val.size(); ++j) {
|
|
if (!is_stmt<identifier>(tuple->val[j])) {
|
|
throw std::runtime_error("Cannot unpack non-identifier type: " + tuple->val[j]->type());
|
|
}
|
|
auto id = cast_stmt<identifier>(tuple->val[j])->val;
|
|
ctx.set_val(id, c_arr[j]);
|
|
}
|
|
};
|
|
|
|
} else {
|
|
throw std::runtime_error("Invalid loop variable(s): " + loopvar->type());
|
|
}
|
|
|
|
if (select_expr && test_expr) {
|
|
scope_update_fn(loop_scope);
|
|
value test_val = test_expr->execute(loop_scope);
|
|
if (!test_val->as_bool()) {
|
|
continue;
|
|
}
|
|
}
|
|
JJ_DEBUG("For loop: adding item type %s at index %zu", current->type().c_str(), i);
|
|
filtered_items.push_back(current);
|
|
scope_update_fns.push_back(scope_update_fn);
|
|
}
|
|
JJ_DEBUG("For loop: %zu items after filtering", filtered_items.size());
|
|
|
|
auto result = mk_val<value_array>();
|
|
|
|
bool noIteration = true;
|
|
for (size_t i = 0; i < filtered_items.size(); i++) {
|
|
JJ_DEBUG("For loop iteration %zu/%zu", i + 1, filtered_items.size());
|
|
value_object loop_obj = mk_val<value_object>();
|
|
loop_obj->has_builtins = false; // loop object has no builtins
|
|
loop_obj->insert("index", mk_val<value_int>(i + 1));
|
|
loop_obj->insert("index0", mk_val<value_int>(i));
|
|
loop_obj->insert("revindex", mk_val<value_int>(filtered_items.size() - i));
|
|
loop_obj->insert("revindex0", mk_val<value_int>(filtered_items.size() - i - 1));
|
|
loop_obj->insert("first", mk_val<value_bool>(i == 0));
|
|
loop_obj->insert("last", mk_val<value_bool>(i == filtered_items.size() - 1));
|
|
loop_obj->insert("length", mk_val<value_int>(filtered_items.size()));
|
|
loop_obj->insert("previtem", i > 0 ? filtered_items[i - 1] : mk_val<value_undefined>("previtem"));
|
|
loop_obj->insert("nextitem", i < filtered_items.size() - 1 ? filtered_items[i + 1] : mk_val<value_undefined>("nextitem"));
|
|
scope.set_val("loop", loop_obj);
|
|
scope_update_fns[i](scope);
|
|
try {
|
|
for (auto & stmt : body) {
|
|
value val = stmt->execute(scope);
|
|
result->push_back(val);
|
|
}
|
|
} catch (const continue_statement::signal &) {
|
|
continue;
|
|
} catch (const break_statement::signal &) {
|
|
break;
|
|
}
|
|
noIteration = false;
|
|
}
|
|
|
|
JJ_DEBUG("For loop complete, total iterations: %zu", filtered_items.size());
|
|
if (noIteration) {
|
|
for (auto & stmt : default_block) {
|
|
value val = stmt->execute(ctx);
|
|
result->push_back(val);
|
|
}
|
|
}
|
|
|
|
// convert to string parts
|
|
value_string str = mk_val<value_string>();
|
|
gather_string_parts_recursive(result, str);
|
|
return str;
|
|
}
|
|
|
|
value set_statement::execute_impl(context & ctx) {
|
|
auto rhs = val ? val->execute(ctx) : exec_statements(body, ctx);
|
|
|
|
if (is_stmt<identifier>(assignee)) {
|
|
// case: {% set my_var = value %}
|
|
auto var_name = cast_stmt<identifier>(assignee)->val;
|
|
JJ_DEBUG("Setting global variable '%s' with value type %s", var_name.c_str(), rhs->type().c_str());
|
|
ctx.set_val(var_name, rhs);
|
|
|
|
} else if (is_stmt<tuple_literal>(assignee)) {
|
|
// case: {% set a, b = value %}
|
|
auto tuple = cast_stmt<tuple_literal>(assignee);
|
|
if (!is_val<value_array>(rhs)) {
|
|
throw std::runtime_error("Cannot unpack non-iterable type in set: " + rhs->type());
|
|
}
|
|
auto & arr = rhs->as_array();
|
|
if (arr.size() != tuple->val.size()) {
|
|
throw std::runtime_error(std::string("Too ") + (tuple->val.size() > arr.size() ? "few" : "many") + " items to unpack in set");
|
|
}
|
|
for (size_t i = 0; i < tuple->val.size(); ++i) {
|
|
auto & elem = tuple->val[i];
|
|
if (!is_stmt<identifier>(elem)) {
|
|
throw std::runtime_error("Cannot unpack to non-identifier in set: " + elem->type());
|
|
}
|
|
auto var_name = cast_stmt<identifier>(elem)->val;
|
|
ctx.set_val(var_name, arr[i]);
|
|
}
|
|
|
|
} else if (is_stmt<member_expression>(assignee)) {
|
|
// case: {% set ns.my_var = value %}
|
|
auto member = cast_stmt<member_expression>(assignee);
|
|
if (member->computed) {
|
|
throw std::runtime_error("Cannot assign to computed member");
|
|
}
|
|
if (!is_stmt<identifier>(member->property)) {
|
|
throw std::runtime_error("Cannot assign to member with non-identifier property");
|
|
}
|
|
auto prop_name = cast_stmt<identifier>(member->property)->val;
|
|
|
|
value object = member->object->execute(ctx);
|
|
if (!is_val<value_object>(object)) {
|
|
throw std::runtime_error("Cannot assign to member of non-object");
|
|
}
|
|
auto obj_ptr = cast_val<value_object>(object);
|
|
JJ_DEBUG("Setting object property '%s' with value type %s", prop_name.c_str(), rhs->type().c_str());
|
|
obj_ptr->insert(prop_name, rhs);
|
|
|
|
} else {
|
|
throw std::runtime_error("Invalid LHS inside assignment expression: " + assignee->type());
|
|
}
|
|
return mk_val<value_undefined>();
|
|
}
|
|
|
|
value macro_statement::execute_impl(context & ctx) {
|
|
if (!is_stmt<identifier>(this->name)) {
|
|
throw std::runtime_error("Macro name must be an identifier");
|
|
}
|
|
std::string name = cast_stmt<identifier>(this->name)->val;
|
|
|
|
const func_handler func = [this, name, &ctx](const func_args & args) -> value {
|
|
size_t expected_count = this->args.size();
|
|
size_t input_count = args.count();
|
|
|
|
JJ_DEBUG("Invoking macro '%s' with %zu input arguments (expected %zu)", name.c_str(), input_count, expected_count);
|
|
context macro_ctx(ctx); // new scope for macro execution
|
|
|
|
// bind parameters
|
|
for (size_t i = 0; i < expected_count; ++i) {
|
|
if (i < input_count) {
|
|
if (is_stmt<identifier>(this->args[i])) {
|
|
// normal parameter
|
|
std::string param_name = cast_stmt<identifier>(this->args[i])->val;
|
|
value param_value = args.get_kwarg_or_pos(param_name, i);
|
|
JJ_DEBUG(" Binding parameter '%s' to argument of type %s", param_name.c_str(), param_value->type().c_str());
|
|
macro_ctx.set_val(param_name, param_value);
|
|
} else if (is_stmt<keyword_argument_expression>(this->args[i])) {
|
|
// default argument used as normal parameter
|
|
auto kwarg = cast_stmt<keyword_argument_expression>(this->args[i]);
|
|
if (!is_stmt<identifier>(kwarg->key)) {
|
|
throw std::runtime_error("Keyword argument key must be an identifier in macro '" + name + "'");
|
|
}
|
|
std::string param_name = cast_stmt<identifier>(kwarg->key)->val;
|
|
value param_value = args.get_kwarg_or_pos(param_name, i);
|
|
JJ_DEBUG(" Binding parameter '%s' to argument of type %s", param_name.c_str(), param_value->type().c_str());
|
|
macro_ctx.set_val(param_name, param_value);
|
|
} else {
|
|
throw std::runtime_error("Invalid parameter type in macro '" + name + "'");
|
|
}
|
|
} else {
|
|
auto & default_arg = this->args[i];
|
|
if (is_stmt<keyword_argument_expression>(default_arg)) {
|
|
auto kwarg = cast_stmt<keyword_argument_expression>(default_arg);
|
|
if (!is_stmt<identifier>(kwarg->key)) {
|
|
throw std::runtime_error("Keyword argument key must be an identifier in macro '" + name + "'");
|
|
}
|
|
std::string param_name = cast_stmt<identifier>(kwarg->key)->val;
|
|
JJ_DEBUG(" Binding parameter '%s' to default argument of type %s", param_name.c_str(), kwarg->val->type().c_str());
|
|
macro_ctx.set_val(param_name, kwarg->val->execute(ctx));
|
|
} else {
|
|
throw std::runtime_error("Not enough arguments provided to macro '" + name + "'");
|
|
}
|
|
//std::string param_name = cast_stmt<identifier>(default_args[i])->val;
|
|
//JJ_DEBUG(" Binding parameter '%s' to default", param_name.c_str());
|
|
//macro_ctx.var[param_name] = default_args[i]->execute(ctx);
|
|
}
|
|
}
|
|
|
|
// execute macro body
|
|
JJ_DEBUG("Executing macro '%s' body with %zu statements", name.c_str(), this->body.size());
|
|
auto res = exec_statements(this->body, macro_ctx);
|
|
JJ_DEBUG("Macro '%s' execution complete, result: %s", name.c_str(), res->val_str.str().c_str());
|
|
return res;
|
|
};
|
|
|
|
JJ_DEBUG("Defining macro '%s' with %zu parameters", name.c_str(), args.size());
|
|
ctx.set_val(name, mk_val<value_func>(name, func));
|
|
return mk_val<value_undefined>();
|
|
}
|
|
|
|
value member_expression::execute_impl(context & ctx) {
|
|
value object = this->object->execute(ctx);
|
|
|
|
value property;
|
|
if (this->computed) {
|
|
// syntax: obj[expr]
|
|
JJ_DEBUG("Member expression, computing property type %s", this->property->type().c_str());
|
|
|
|
int64_t arr_size = 0;
|
|
if (is_val<value_array>(object)) {
|
|
arr_size = object->as_array().size();
|
|
} else if (is_val<value_string>(object)) {
|
|
arr_size = object->as_string().length();
|
|
}
|
|
|
|
if (is_stmt<slice_expression>(this->property)) {
|
|
auto s = cast_stmt<slice_expression>(this->property);
|
|
value start_val = s->start_expr ? s->start_expr->execute(ctx) : mk_val<value_int>(0);
|
|
value stop_val = s->stop_expr ? s->stop_expr->execute(ctx) : mk_val<value_int>(arr_size);
|
|
value step_val = s->step_expr ? s->step_expr->execute(ctx) : mk_val<value_int>(1);
|
|
|
|
// translate to function call: obj.slice(start, stop, step)
|
|
JJ_DEBUG("Member expression is a slice: start %s, stop %s, step %s",
|
|
start_val->as_repr().c_str(),
|
|
stop_val->as_repr().c_str(),
|
|
step_val->as_repr().c_str());
|
|
auto slice_func = try_builtin_func(ctx, "slice", object);
|
|
func_args args(ctx);
|
|
args.push_back(start_val);
|
|
args.push_back(stop_val);
|
|
args.push_back(step_val);
|
|
return slice_func->invoke(args);
|
|
} else {
|
|
property = this->property->execute(ctx);
|
|
}
|
|
} else {
|
|
// syntax: obj.prop
|
|
if (!is_stmt<identifier>(this->property)) {
|
|
throw std::runtime_error("Static member property must be an identifier");
|
|
}
|
|
property = mk_val<value_string>(cast_stmt<identifier>(this->property)->val);
|
|
std::string prop = property->as_string().str();
|
|
JJ_DEBUG("Member expression, object type %s, static property '%s'", object->type().c_str(), prop.c_str());
|
|
|
|
// behavior of jinja2: obj having prop as a built-in function AND 'prop', as an object key,
|
|
// then obj.prop returns the built-in function, not the property value.
|
|
// while obj['prop'] returns the property value.
|
|
// example: {"obj": {"items": 123}} -> obj.items is the built-in function, obj['items'] is 123
|
|
|
|
value val = try_builtin_func(ctx, prop, object, true);
|
|
if (!is_val<value_undefined>(val)) {
|
|
return val;
|
|
}
|
|
// else, fallthrough to normal property access below
|
|
}
|
|
|
|
JJ_DEBUG("Member expression on object type %s, property type %s", object->type().c_str(), property->type().c_str());
|
|
value val = mk_val<value_undefined>("object_property");
|
|
|
|
if (property->is_undefined()) {
|
|
JJ_DEBUG("%s", "Member expression property is undefined, returning undefined");
|
|
return val;
|
|
}
|
|
|
|
ensure_key_type_allowed(property);
|
|
|
|
if (is_val<value_undefined>(object)) {
|
|
JJ_DEBUG("%s", "Accessing property on undefined object, returning undefined");
|
|
return val;
|
|
|
|
} else if (is_val<value_object>(object)) {
|
|
auto key = property->as_string().str();
|
|
val = object->at(property, val);
|
|
if (is_val<value_undefined>(val)) {
|
|
val = try_builtin_func(ctx, key, object, true);
|
|
}
|
|
JJ_DEBUG("Accessed property '%s' value, got type: %s", key.c_str(), val->type().c_str());
|
|
|
|
} else if (is_val<value_array>(object) || is_val<value_string>(object)) {
|
|
if (is_val<value_int>(property)) {
|
|
int64_t index = property->as_int();
|
|
JJ_DEBUG("Accessing %s index %d", object->type().c_str(), (int)index);
|
|
if (is_val<value_array>(object)) {
|
|
auto & arr = object->as_array();
|
|
if (index < 0) {
|
|
index += static_cast<int64_t>(arr.size());
|
|
}
|
|
if (index >= 0 && index < static_cast<int64_t>(arr.size())) {
|
|
val = arr[index];
|
|
}
|
|
} else { // value_string
|
|
auto str = object->as_string().str();
|
|
if (index >= 0 && index < static_cast<int64_t>(str.size())) {
|
|
val = mk_val<value_string>(std::string(1, str[index]));
|
|
}
|
|
}
|
|
|
|
} else if (is_val<value_string>(property)) {
|
|
auto key = property->as_string().str();
|
|
JJ_DEBUG("Accessing %s built-in '%s'", is_val<value_array>(object) ? "array" : "string", key.c_str());
|
|
val = try_builtin_func(ctx, key, object, true);
|
|
|
|
} else {
|
|
throw std::runtime_error("Cannot access property with non-string/non-number: got " + property->type());
|
|
}
|
|
} else {
|
|
if (!is_val<value_string>(property)) {
|
|
throw std::runtime_error("Cannot access property with non-string: got " + property->type());
|
|
}
|
|
auto key = property->as_string().str();
|
|
val = try_builtin_func(ctx, key, object, true);
|
|
}
|
|
|
|
if (ctx.is_get_stats && val && object && property) {
|
|
value_t::stats_t::mark_used(val);
|
|
value_t::stats_t::mark_used(object);
|
|
value_t::stats_t::mark_used(property);
|
|
if (is_val<value_int>(property)) {
|
|
object->stats.ops.insert("array_access");
|
|
} else if (is_val<value_string>(property)) {
|
|
object->stats.ops.insert("object_access");
|
|
}
|
|
}
|
|
|
|
return val;
|
|
}
|
|
|
|
value call_expression::execute_impl(context & ctx) {
|
|
// gather arguments
|
|
func_args args(ctx);
|
|
for (auto & arg_stmt : this->args) {
|
|
auto arg_val = arg_stmt->execute(ctx);
|
|
JJ_DEBUG(" Argument type: %s", arg_val->type().c_str());
|
|
args.push_back(arg_val);
|
|
}
|
|
// execute callee
|
|
value callee_val = callee->execute(ctx);
|
|
if (!is_val<value_func>(callee_val)) {
|
|
throw std::runtime_error("Callee is not a function: got " + callee_val->type());
|
|
}
|
|
auto * callee_func = cast_val<value_func>(callee_val);
|
|
JJ_DEBUG("Calling function '%s' with %zu arguments", callee_func->name.c_str(), args.count());
|
|
return callee_func->invoke(args);
|
|
}
|
|
|
|
value keyword_argument_expression::execute_impl(context & ctx) {
|
|
if (!is_stmt<identifier>(key)) {
|
|
throw std::runtime_error("Keyword argument key must be identifiers");
|
|
}
|
|
|
|
std::string k = cast_stmt<identifier>(key)->val;
|
|
JJ_DEBUG("Keyword argument expression key: %s, value: %s", k.c_str(), val->type().c_str());
|
|
|
|
value v = val->execute(ctx);
|
|
JJ_DEBUG("Keyword argument value executed, type: %s", v->type().c_str());
|
|
|
|
return mk_val<value_kwarg>(k, v);
|
|
}
|
|
|
|
} // namespace jinja
|