The current project is an occam compiler. I'd like you to explain one thing about its functionality: there is an #include keyword allowed in source files input to the compiler. For example you can see this used in line 1 of the file ./modules/course/examples/hello_world.occ. I'd like to learn how that include feature is implemented. I've looked at one of the "modules" (which are directories with occam source files and subdirectories) to see how it might work. For example the "course" module is at ./modules/course. I don't think the include feature simply processes all source files under the module root directory. I suspect it is more complex than that. I see files in the directory structure that might be involved. They look like Python source files. For example I see one at ./modules/course/SConscript . My hunch is that the compiler looks for these files and perhaps executes them to get the actual set of source files to include for a given module, but I'm not sure. Can you look into this and let me know how it works please? By the way the compiler itsef is written in C.
The user wants to understand how the #INCLUDE feature works in this occam compiler. Let me explore the codebase to understand the mechanism. Let me start by looking at the example file, the SConscript files, and the compiler source code to trace how includes are resolved.
Let me explore the codebase to trace how #INCLUDE works, starting from the example file and working through the build system and compiler source.
{
"file_path": "/home/david/projects/codeassociates/kroc/modules/course/examples/hello_world.occ"
} 1→#INCLUDE "course.module"
2→
3→PROC hello.world (CHAN BYTE keyboard?, screen!, error!)
4→ --{{{
5→ out.string ("Hello World*c*n", 0, screen!)
6→ --}}}
7→:
8→
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"file_path": "/home/david/projects/codeassociates/kroc/modules/course/SConscript"
} 1→import os
2→
3→dirs = Split('libsrc answers examples')
4→
5→for d in dirs:
6→ SConscript(os.path.join(d, 'SConscript'))
7→
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
/home/david/projects/codeassociates/kroc/modules/course/answers/README /home/david/projects/codeassociates/kroc/modules/course/answers/SConscript /home/david/projects/codeassociates/kroc/modules/course/answers/configure.ac /home/david/projects/codeassociates/kroc/modules/course/answers/q1.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q2-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q2.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q1-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/Makefile.am /home/david/projects/codeassociates/kroc/modules/course/answers/q3-1.tce /home/david/projects/codeassociates/kroc/modules/course/README /home/david/projects/codeassociates/kroc/modules/course/SConscript /home/david/projects/codeassociates/kroc/modules/course/answers/q3.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-adam-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q6-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q4-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q5-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q4.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-eric-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q6.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-adam.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-nick-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q7-jonathan2.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-jonathan-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q7-jonathan.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-jonathan2-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q7-fred-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q6-sort.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-fred.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q7-eric.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-nick-2.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q5.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q6-sort-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q7-richard.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-nick.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-nick-3.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q7.occ /home/david/projects/codeassociates/kroc/modules/course/answers/q7-richard-1.tce /home/david/projects/codeassociates/kroc/modules/course/answers/q7-richard-2.tce /home/david/projects/codeassociates/kroc/modules/course/etc/README /home/david/projects/codeassociates/kroc/modules/course/etc/reading.list /home/david/projects/codeassociates/kroc/modules/course/examples/bar.occ /home/david/projects/codeassociates/kroc/modules/course/examples/hello_seq_world.occ /home/david/projects/codeassociates/kroc/modules/course/examples/casting.occ /home/david/projects/codeassociates/kroc/modules/course/examples/SConscript /home/david/projects/codeassociates/kroc/modules/course/examples/commstime.occ /home/david/projects/codeassociates/kroc/modules/course/examples/configure.ac /home/david/projects/codeassociates/kroc/modules/course/examples/demo.occ /home/david/projects/codeassociates/kroc/modules/course/examples/hello_raw_world.occ /home/david/projects/codeassociates/kroc/modules/course/examples/sort_pump.occ /home/david/projects/codeassociates/kroc/modules/course/examples/README.txt /home/david/projects/codeassociates/kroc/modules/course/examples/double_echo.occ /home/david/projects/codeassociates/kroc/modules/course/examples/Makefile.am /home/david/projects/codeassociates/kroc/modules/course/examples/cast.occ /home/david/projects/codeassociates/kroc/modules/course/examples/sort_inside_2.occ /home/david/projects/codeassociates/kroc/modules/course/examples/README_kroc.txt /home/david/projects/codeassociates/kroc/modules/course/examples/sort_inside.occ /home/david/projects/codeassociates/kroc/modules/course/examples/echoing.occ /home/david/projects/codeassociates/kroc/modules/course/examples/hello_world.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/bugs.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/configure.ac /home/david/projects/codeassociates/kroc/modules/course/exercises/README.txt /home/david/projects/codeassociates/kroc/modules/course/exercises/cylons.doc /home/david/projects/codeassociates/kroc/modules/course/exercises/Makefile.am /home/david/projects/codeassociates/kroc/modules/course/examples/sort_pump_2.occ /home/david/projects/codeassociates/kroc/modules/course/examples/test_utils.occ /home/david/projects/codeassociates/kroc/modules/course/examples/test_bell.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/cylons.pdf /home/david/projects/codeassociates/kroc/modules/course/exercises/cylons.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/extra-exercises.pdf /home/david/projects/codeassociates/kroc/modules/course/exercises/more-exercises.doc /home/david/projects/codeassociates/kroc/modules/course/exercises/e1.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/extra-exercises.doc /home/david/projects/codeassociates/kroc/modules/course/exercises/early-exercises.pdf /home/david/projects/codeassociates/kroc/modules/course/exercises/early-exercises.doc /home/david/projects/codeassociates/kroc/modules/course/exercises/nn_data.txt /home/david/projects/codeassociates/kroc/modules/course/exercises/phils-plan.txt /home/david/projects/codeassociates/kroc/modules/course/exercises/phils-plan.pdf /home/david/projects/codeassociates/kroc/modules/course/exercises/phils-plan-large.doc /home/david/projects/codeassociates/kroc/modules/course/exercises/phils-plan.doc /home/david/projects/codeassociates/kroc/modules/course/exercises/nn_small.txt /home/david/projects/codeassociates/kroc/modules/course/exercises/q1.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/nn_tiny.txt /home/david/projects/codeassociates/kroc/modules/course/exercises/more-exercises.pdf /home/david/projects/codeassociates/kroc/modules/course/exercises/phils-plan-large.pdf /home/david/projects/codeassociates/kroc/modules/course/exercises/q3.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/q4.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/small_results_sorted.txt /home/david/projects/codeassociates/kroc/modules/course/exercises/q6.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/santa-exercise.doc /home/david/projects/codeassociates/kroc/modules/course/exercises/q7.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/small_results.txt /home/david/projects/codeassociates/kroc/modules/course/exercises/santa-exercise.pdf /home/david/projects/codeassociates/kroc/modules/course/exercises/robots-sei.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/q2.occ /home/david/projects/codeassociates/kroc/modules/course/exercises/q5.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/README /home/david/projects/codeassociates/kroc/modules/course/libsrc/Makefile.am /home/david/projects/codeassociates/kroc/modules/course/libsrc/libcourse-nets.3 /home/david/projects/codeassociates/kroc/modules/course/libsrc/utils.doc (Results are truncated. Consider using a more specific path or pattern.)
/home/david/projects/codeassociates/kroc/modules/course/libsrc/README /home/david/projects/codeassociates/kroc/modules/course/libsrc/Makefile.am /home/david/projects/codeassociates/kroc/modules/course/libsrc/libcourse-nets.3 /home/david/projects/codeassociates/kroc/modules/course/libsrc/utils.doc /home/david/projects/codeassociates/kroc/modules/course/libsrc/libcourse-cycles.3 /home/david/projects/codeassociates/kroc/modules/course/libsrc/float_io.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/consts.inc /home/david/projects/codeassociates/kroc/modules/course/libsrc/random.doc /home/david/projects/codeassociates/kroc/modules/course/libsrc/SConscript /home/david/projects/codeassociates/kroc/modules/course/libsrc/configure.ac /home/david/projects/codeassociates/kroc/modules/course/libsrc/string.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/stringbuf.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/utils.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/file_in.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/random.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/demo_nets.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/shared_screen.inc /home/david/projects/codeassociates/kroc/modules/course/libsrc/demo_cycles.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/shared_screen.occ /home/david/projects/codeassociates/kroc/modules/course/libsrc/libcourse-utils.3
{
"pattern": "course\\.module",
"path": "/home/david/projects/codeassociates/kroc/modules/course"
}Found 48 files modules/course/libsrc/configure.ac modules/course/libsrc/consts.inc modules/course/libsrc/demo_cycles.occ modules/course/libsrc/demo_nets.occ modules/course/exercises/q2.occ modules/course/exercises/q3.occ modules/course/exercises/q4.occ modules/course/exercises/q5.occ modules/course/exercises/q6.occ modules/course/exercises/q7.occ modules/course/exercises/robots-sei.occ modules/course/exercises/q1.occ modules/course/exercises/cylons.occ modules/course/exercises/e1.occ modules/course/examples/sort_pump_2.occ modules/course/examples/test_bell.occ modules/course/examples/test_utils.occ modules/course/exercises/bugs.occ modules/course/exercises/configure.ac modules/course/answers/q7-nick.occ modules/course/answers/q7-richard.occ modules/course/answers/q7.occ modules/course/examples/bar.occ modules/course/examples/cast.occ modules/course/examples/casting.occ modules/course/examples/commstime.occ modules/course/examples/configure.ac modules/course/examples/demo.occ modules/course/examples/double_echo.occ modules/course/examples/echoing.occ modules/course/examples/hello_world.occ modules/course/examples/README.txt modules/course/examples/sort_inside_2.occ modules/course/examples/sort_inside.occ modules/course/examples/sort_pump.occ modules/course/answers/q3.occ modules/course/answers/q4.occ modules/course/answers/q5.occ modules/course/answers/q6-sort.occ modules/course/answers/q6.occ modules/course/answers/q7-adam.occ modules/course/answers/q7-eric.occ modules/course/answers/q7-fred.occ modules/course/answers/q7-jonathan.occ modules/course/answers/q7-jonathan2.occ modules/course/answers/configure.ac modules/course/answers/q1.occ modules/course/answers/q2.occ
{
"pattern": "\\.module",
"path": "/home/david/projects/codeassociates/kroc/modules/course/libsrc"
}Found 5 files modules/course/libsrc/configure.ac modules/course/libsrc/consts.inc modules/course/libsrc/demo_cycles.occ modules/course/libsrc/demo_nets.occ modules/course/libsrc/shared_screen.occ
{
"file_path": "/home/david/projects/codeassociates/kroc/modules/course/libsrc/SConscript"
} 1→Import('env')
2→local = env.Clone()
3→
4→course_lib_srcs = Split('''
5→ utils.occ
6→ string.occ
7→ demo_cycles.occ
8→ demo_nets.occ
9→ file_in.occ
10→ float_io.occ
11→ random.occ
12→ ''')
13→
14→shared_screen_lib_srcs = Split('''
15→ shared_screen.occ
16→ ''')
17→shared_screen_lib_objs = \
18→ [local.OccamObject(f, INCPATH='.') for f in shared_screen_lib_srcs]
19→
20→
21→
22→course_lib = local.OccamLibrary(
23→ 'course.lib',
24→ course_lib_srcs,
25→ INCPATH='.',
26→ OCCBUILDFLAGS='--include consts.inc')
27→
28→def mk_shared_screen(lib_name):
29→ return local.OccamLibrary(
30→ lib_name,
31→ shared_screen_lib_objs,
32→ INCPATH='.',
33→ OCCBUILDFLAGS='--need course --include shared_screen.inc')
34→
35→sharedscreen_lib = mk_shared_screen('shared_screen.lib')
36→# Build ss.lib too for backwards compatibility.
37→mk_shared_screen('ss.lib')
38→
39→# FIXME: Autodepend???
40→local.Depends('shared_screen.tce', course_lib)
41→local.Depends('demo_nets.occ', 'demo_cycles.tce')
42→local.Depends('string.occ', 'utils.tce')
43→
44→
45→env['OCCLIBS']['course'] = dict(
46→ dep=course_lib,
47→ inc=Dir('.'))
48→env['OCCLIBS']['shared_screen'] = dict(
49→ dep=sharedscreen_lib,
50→ inc=Dir('.'))
51→
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"file_path": "/home/david/projects/codeassociates/kroc/modules/course/libsrc/configure.ac"
} 1→AC_INIT([course], [1.0], [kroc-bugs@kent.ac.uk])
2→AC_CANONICAL_SYSTEM
3→
4→AM_INIT_AUTOMAKE
5→
6→dnl Checks for programs.
7→OCCAM_OCCBUILD
8→
9→OCCAM_PROVIDE(course.module shared_screen.module ss.module)
10→
11→AC_OUTPUT([Makefile])
12→
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"file_path": "/home/david/projects/codeassociates/kroc/modules/course/libsrc/consts.inc"
} 1→--
2→-- Constants
3→-- Copyright (C) 1984 P.H. Welch
4→--
5→-- This library is free software; you can redistribute it and/or
6→-- modify it under the terms of the GNU Lesser General Public
7→-- License as published by the Free Software Foundation; either
8→-- version 2 of the License, or (at your option) any later version.
9→--
10→-- This library is distributed in the hope that it will be useful,
11→-- but WITHOUT ANY WARRANTY; without even the implied warranty of
12→-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
13→-- Lesser General Public License for more details.
14→--
15→-- You should have received a copy of the GNU Lesser General Public
16→-- License along with this library; if not, write to the Free Software
17→-- Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
18→--
19→
20→--** Course library.
21→--
22→-- The course library provides various simple utilities for occam programmers.
23→--
24→-- Strings (as handled by this library) are [@code BYTE] arrays left-justified
25→-- with trailing [@ref NULL]s. It is usually preferably to use mobile byte
26→-- arrays ([@code MOBILE []BYTE]) to represent strings in modern occam-pi
27→-- programs; the string features in this library exist for backwards
28→-- compatibility.
29→--
30→-- To use this library:
31→--
32→-- [@code
33→-- #INCLUDE "course.module"]
34→--
35→-- @module course
36→
37→--*{{{ ASCII Character values.
38→VAL BYTE NULL IS 0: --* ASCII NUL
39→VAL BYTE BELL IS 7: --* ASCII BEL - terminal bell
40→VAL BYTE BACK IS 8: --* ASCII BS - backspace key
41→VAL BYTE ESCAPE IS 27: --* ASCII ESC - escape key
42→VAL BYTE DELETE IS 127: --* ASCII DEL - delete key
43→--*}}}
44→
45→--{{{ special constants
46→VAL BYTE FLUSH IS 255: --* Flush output buffer
47→VAL BYTE END.OF.FILE IS 255: --* End of file
48→--}}}
49→
50→--*{{{ COMPARE.STRING Return values from [@ref compare.string].
51→VAL INT string.less IS -1: --* Left less than right
52→VAL INT string.same IS 0: --* Left and right the same
53→VAL INT string.more IS 1: --* Left more than right
54→--*}}}
55→
56→
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
Key finding: The configure.ac file has OCCAM_PROVIDE(course.module shared_screen.module ss.module). This is interesting - it suggests there's an autoconf macro OCCAM_PROVIDE that registers .module files. Let me now look at the compiler source to understand how #INCLUDE "course.module" is handled. Let me also look for the OCCAM_PROVIDE macro and the occbuild tool.
Now let me dig into the compiler and build tools to understand how .module files are resolved.
{
"subagent_type": "Explore",
"prompt": "I'm investigating how `#INCLUDE \"course.module\"` works in this occam compiler project. I need to understand the full chain from the source directive to file resolution.\n\nPlease do the following searches (be very thorough):\n\n1. Find where `OCCAM_PROVIDE` is defined - it's likely an autoconf/m4 macro. Search for it in .m4 files or aclocal files.\n\n2. Find the `occbuild` tool - it appears to be invoked in the SConscript build files. Look for it as a script (probably Python or shell).\n\n3. In the C compiler source, find where `#INCLUDE` directives are parsed and how the compiler resolves file paths (especially `.module` files). Search for strings like \"INCLUDE\", \".module\", \"include\" in the C source files.\n\n4. Look for any search path mechanism - how does the compiler know where to find \"course.module\"? Look for things like INCPATH, include path, search path, -I flags, etc.\n\n5. Look for any code that generates or processes `.module` files - are they actual files on disk, or are they virtual/generated?\n\nSearch broadly across the whole project."
}[
{
"type": "text",
"text": "Perfect! Now I have all the pieces. Let me create a comprehensive summary with diagrams of the flow:\n\n## Summary Report: How `#INCLUDE \"course.module\"` Works\n\nI've thoroughly investigated the occam compiler project and traced the complete chain from the `#INCLUDE` directive to file resolution. Here's the full mechanism:\n\n### 1. **OCCAM_PROVIDE Autoconf Macro** \n**Location:** `/home/david/projects/codeassociates/kroc/m4/occbuild.m4` (lines 217-245)\n\nThe `OCCAM_PROVIDE` macro is defined in the autoconf configuration file. It records which modules are provided by a package:\n\n```\nOCCAM_PROVIDE(FILES, [SUBDIRECTORY], [DEPENDENCIES], [CONDITION])\n```\n\n**Example usage** in `/home/david/projects/codeassociates/kroc/modules/course/libsrc/configure.ac`:\n```\nOCCAM_PROVIDE(course.module shared_screen.module ss.module)\n```\n\n**How it works:**\n- Maintains a file called `$KROC_BUILD_ROOT/in-tree-modules`\n- Each line contains: `<module-name> <directory> <dependencies>`\n- If a module can't be built (CONDITION fails), the line becomes: `<module-name> -`\n- This file is updated and sorted during the build process\n\n### 2. **occbuild Tool - The Module Registration & Resolution System**\n**Locations:**\n- Python implementation: `/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in` (1254 lines)\n- Autoconf macro: `/home/david/projects/codeassociates/kroc/m4/occbuild.m4`\n- SCons tool: `/home/david/projects/codeassociates/kroc/site_scons/site_tools/occbuild.py`\n\n**Key occbuild functions relevant to module resolution:**\n\n**a) `make_module()` function (lines 290-317):** Generates `.module` files\n```python\ndef make_module(name, custom):\n ofn = name + \".module\"\n f = open(ofn, \"w\")\n f.write('-- Include file for \"%s\" module\\n' % name)\n # ... header guards and includes ...\n do_list('#INCLUDE \"%s.module\"\\n', needs) # Include dependencies\n do_list('#INCLUDE \"%s\"\\n', includes_before) # Custom includes\n custom(f) # Toolchain-specific content\n do_list('#INCLUDE \"%s\"\\n', includes_after) # More custom includes\n```\n\n**b) `search_path` mechanism:** The compiler searches multiple directories for modules, built from:\n- In-tree dependencies (lines 245-252)\n- Environment variables like `ISEARCH` (line 256)\n- Search directories built by build system\n\n**c) Module resolution in `KRoCToolchain.library()` (lines 490-496):**\n```python\nfull_path = search_path + capture(programs[\"kroc\"] + [\"--incpath\"]).strip().split(\":\")\nfor need in need_mods:\n fn = find_in_path(\"%s.module\" % need, full_path)\n if fn is None:\n die(\"Cannot find module \", need, \" in search path...\")\n```\n\n### 3. **Compiler Source Code - #INCLUDE Parsing**\n**Key files:**\n- Parser: `/home/david/projects/codeassociates/kroc/tools/occ21/fe/syn2.c` (lines 6083-6167)\n- Lexer: `/home/david/projects/codeassociates/kroc/tools/occ21/fe/lex1.c`\n\n**The parsing flow:**\n\n**a) `rfile()` function in syn2.c (lines 6086-6167):**\n- Handles `#INCLUDE`, `#USE`, `#IMPORT`, `#SC` directives\n- Determines the file mode:\n - If ends in `.co`: `LEX_CSOURCE` (C source file)\n - Otherwise: `LEX_SOURCE` (occam source)\n - If `#USE` or `#IMPORT`: `LEX_LIB` (library/module file)\n- Calls `open_file(litbuf, mode, indent)` to find and open the file\n\n**b) `open_file()` function in lex1.c (lines 1234-1280):**\n```c\nswitch (mode) {\n case LEX_SOURCE:\n case LEX_CSOURCE:\n fptr = popen_relative(name, current_fe_data->fe_pathname, NULL, \"r\", ...);\n break;\n case LEX_SC:\n case LEX_LIB: // Module/library file handling\n case LEX_STDLIB:\n fptr = open_descfile(name, mode);\n break;\n}\n```\n\n### 4. **File Resolution - The Actual Search Mechanism**\n\n**a) `open_descfile()` in objrdr.c (lines 489-534):**\n- Opens descriptor files (`.lib` files which are binary TCOFF format)\n- Extracts module information from the compiled library\n- Uses `popen_relative()` for the actual file search\n\n**b) `popen_relative()` in fnilib/popen.c (lines 130-200):**\nThe core file search engine that:\n1. Checks if the file is in the current directory\n2. Extracts search path from environment variable (e.g., `ISEARCH`)\n3. Iterates through all directories in the search path\n4. Returns the first matching file found\n\n**Search path construction (in occbuild.py, lines 239-259):**\n```python\ndef build_default_search_path(tc_search_path):\n global search_path\n if in_tree is not None:\n search_path += [\n in_tree + \"/modules/inmoslibs/libsrc/forall\",\n in_tree + \"/tvm/posix\",\n ]\n else:\n search_path += tc_search_path # Out-of-tree installation paths\n \n old_path = os.getenv(\"ISEARCH\")\n if old_path is not None:\n search_path += old_path.split(':')\n```\n\n### 5. **Complete Resolution Chain for `#INCLUDE \"course.module\"`**\n\n```\nSource file: modules/course/answers/q1.occ\n ↓\nContains: #INCLUDE \"course.module\"\n ↓\nLexer/Parser (syn2.c rfile() function)\n ↓\nDetermines mode = LEX_LIB (for .module files via #USE directive)\n ↓\nCalls open_file(\"course.module\", LEX_LIB, ...)\n ↓\nCalls open_descfile(\"course.module\", LEX_LIB)\n ↓\nCalls popen_relative(\"course.module\", \"ISEARCH\", ...)\n ↓\nSearch in order:\n 1. Current directory\n 2. Directories in ISEARCH environment variable\n 3. Directories added by --search flags (from occbuild)\n 4. In-tree search paths (from in-tree-modules)\n ↓\nFound: modules/course/libsrc/course.module\n ↓\nReads and processes the module file\n```\n\n### 6. **Module File Generation**\n\n**Location:** `modules/course/libsrc/SConscript`\n\nWhen the course library is built:\n\n```python\ncourse_lib = local.OccamLibrary(\n 'course.lib',\n course_lib_srcs,\n INCPATH='.',\n OCCBUILDFLAGS='--include consts.inc')\n```\n\nThis triggers:\n1. Compilation of source files (`utils.occ`, `string.occ`, etc.) to `.tce` files\n2. Creation of `.lib` binary library file\n3. **Generation of `course.module`** via occbuild's `make_module()` function\n\nThe generated `course.module` includes:\n- Guard directives: `#IF NOT (DEFINED (COURSE_MODULE))`\n- Dependencies: `#INCLUDE \"forall.module\"` (if needed)\n- Custom includes: any `--include` flags from build\n- PRAGMA directives for library linking\n\n### 7. **Module Dependency Tracking - in-tree-modules File**\n\nDuring autoconf build configuration, `OCCAM_PROVIDE` updates:\n```\n$KROC_BUILD_ROOT/in-tree-modules\n```\n\nFormat:\n```\ncourse.module /path/to/modules/course/libsrc forall.module shared_screen.module\nshared_screen.module /path/to/modules/course/libsrc course.module\n```\n\nThe `OCCAM_INCLUDE` macro uses this to:\n- Resolve transitive dependencies\n- Add correct `--search` flags to the compiler\n- Ensure all required modules can be found during compilation\n\n### 8. **Build System Integration**\n\n**SCons (site_scons/site_tools/occbuild.py):**\n- Generates commands like: `occbuild --search /path/to/libs --object file.occ`\n- The `--search` flag is passed to the compiler to add directories to INCPATH\n- INCPATH is converted to ISEARCH environment variable for file searching\n\n**Autoconf (m4/occbuild.m4):**\n- `OCCAM_INCLUDE` macro checks for module availability\n- `OCCAM_PROVIDE` macro registers what a package provides\n- Both maintain the `in-tree-modules` file for dependency tracking\n\nThis architecture provides:\n- **Modular compilation**: Libraries can be compiled separately\n- **Flexible search paths**: Can be configured per-build or via environment\n- **Dependency tracking**: Ensures all transitive dependencies are available\n- **Both in-tree and installed builds**: Works with --in-tree flag or standard installation paths"
},
{
"type": "text",
"text": "agentId: a5d6b68 (for resuming to continue this agent's work if needed)\n<usage>total_tokens: 75211\ntool_uses: 44\nduration_ms: 144140</usage>"
}
]Excellent, the agent did a thorough investigation. Let me now verify a few key details by reading some of the critical files myself to make sure the explanation is accurate.
{
"file_path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"offset": 280,
"limit": 50
} 280→ opts = []
281→ if target_os.startswith("darwin"):
282→ opts = ["-dynamiclib", "-read_only_relocs", "suppress"]
283→ elif target_os.startswith("solaris"):
284→ opts = ["-G"]
285→ else:
286→ opts = ["-shared"]
287→
288→ run(programs["cc"] + opts + ["-o", library] + objs + linker_opts + extra_opts)
289→
290→def make_module(name, custom):
291→ """Generate a .module file, calling the provided function to fill in
292→ any toolchain-specific lines."""
293→
294→ ofn = name + ".module"
295→ f = open(ofn, "w")
296→ f.write('-- Include file for "%s" module\n' % name)
297→ f.write('-- %s\n' % autogen_message)
298→ f.write("\n")
299→
300→ guard = ofn.replace("_", ".").upper()
301→ f.write("#IF NOT (DEFINED (%s))\n" % guard)
302→ f.write("#DEFINE %s\n" % guard)
303→ f.write("\n")
304→
305→ def do_list(format, list):
306→ for item in list:
307→ f.write(format % item)
308→ if list != []:
309→ f.write('\n')
310→
311→ do_list('#INCLUDE "%s.module"\n', needs)
312→ do_list('#INCLUDE "%s"\n', includes_before)
313→ custom(f)
314→ do_list('#INCLUDE "%s"\n', includes_after)
315→
316→ f.write("#ENDIF\n")
317→ f.close()
318→
319→def exists(file):
320→ """Does a file or directory exist?"""
321→ return os.access(file, os.F_OK)
322→
323→def make_stub(file):
324→ """Make a stub file to serve as a placeholder."""
325→ f = open(file, "w")
326→ f.close()
327→
328→def is_stub(file):
329→ """Is a file an empty stub?"""
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"file_path": "/home/david/projects/codeassociates/kroc/m4/occbuild.m4",
"offset": 210,
"limit": 50
}210→dnl By default, the dependencies of FILES will be recorded as any files you've 211→dnl checked for using OCCAM_INCLUDE already; if the dependencies are different, 212→dnl pass them as DEPENDENCIES. You can pass "none" as DEPENDENCIES to force the 213→dnl modules to have no dependencies. 214→dnl If the module may only be built sometimes (e.g. if it depends on external 215→dnl libraries that may not be present on some systems), provide a shell command 216→dnl as CONDITION that can be used as an "if" test. 217→dnl OCCAM_PROVIDE(FILES, [SUBDIRECTORY], [DEPENDENCIES], [CONDITION]) 218→AC_DEFUN([OCCAM_PROVIDE], 219→[dnl 220→AC_REQUIRE([OCCAM_IN_TREE]) 221→ 222→if test "x$KROC_BUILD_ROOT" != "x"; then 223→ if test "x$2" = "x"; then 224→ dir=`pwd` 225→ else 226→ dir=`pwd`/$2 227→ fi 228→ if test "x$3" = "x"; then 229→ deps="$OCCAM_INCLUDED" 230→ elif test "x$3" = "xnone"; then 231→ deps="" 232→ else 233→ deps="$3" 234→ fi 235→ if test "x$4" = "x"; then 236→ condition=true 237→ else 238→ condition="$4" 239→ fi 240→ touch $KROC_BUILD_ROOT/in-tree-modules 241→ for file in $1; do 242→ (grep -v "^$file " $KROC_BUILD_ROOT/in-tree-modules; if $condition; then echo "$file $dir $deps"; else echo "$file -"; fi) | sort >$KROC_BUILD_ROOT/in-tree-modules.new 243→ mv -f $KROC_BUILD_ROOT/in-tree-modules.new $KROC_BUILD_ROOT/in-tree-modules 244→ done 245→fi 246→])dnl 247→dnl 248→AC_DEFUN([OCCAM_SWIG], 249→[dnl 250→HAVE_SWIG_OCCAM=no 251→AC_CHECK_PROG(SWIG, swig, swig, no) 252→if test $SWIG != no; then 253→ SWIG_OCCAM="$SWIG -occampi" 254→ AC_SUBST([SWIG_OCCAM]) 255→ dnl 256→ # Check if the -occampi flag is supported 257→ AC_MSG_CHECKING([for SWIG occam-pi support]) 258→ touch conftest.i 259→ if $SWIG_OCCAM -module conftest conftest.i >/dev/null 2>&1; then <system-reminder> Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior. </system-reminder>
{
"file_path": "/home/david/projects/codeassociates/kroc/tools/occ21/fe/syn2.c",
"offset": 6080,
"limit": 90
} 6080→/*}}}*/
6081→/*{{{ PUBLIC BOOL rfile ()*/
6082→/*
6083→ * this is called to handle #INCLUDE, #USE directives.
6084→ * Expecting a string as the next symbol.
6085→ */
6086→PUBLIC BOOL rfile (void)
6087→{
6088→ BOOL entered = FALSE;
6089→ const int s = symb;
6090→ const int indent = lineindent;
6091→
6092→ nextsymb ();
6093→ if (symb != S_STRING) {
6094→ synerr (SYN_MISSING_FILENAME, flocn);
6095→ } else if (s == S_SC) {
6096→ synerr_s (SYN_SC_IS_OBSOLETE, flocn, literalv);
6097→ } else {
6098→ /*{{{ try and open the file */
6099→ char litbuf[MAXSTRING_SIZE];
6100→ int litlen = 0;
6101→ int mode;
6102→ const SOURCEPOSN litlocn = flocn;
6103→
6104→ litlen = strlen (literalv);
6105→ if (litlen >= MAXSTRING_SIZE) {
6106→ litlen = MAXSTRING_SIZE - 1;
6107→ }
6108→ memcpy (litbuf, literalv, litlen);
6109→ litbuf[litlen] = '\0';
6110→
6111→ lex_save_state ();
6112→ nextsymb ();
6113→ while (symb == S_STRING) {
6114→ /* absorb next literal, add to string */
6115→ int blen = strlen (literalv);
6116→
6117→ if ((litlen + blen) >= MAXSTRING_SIZE) {
6118→ blen = (MAXSTRING_SIZE - 1) - litlen;
6119→ }
6120→ if (blen > 0) {
6121→ memcpy (litbuf + litlen, literalv, blen);
6122→ litlen += blen;
6123→ litbuf[litlen] = '\0';
6124→ }
6125→ lex_restore_state ();
6126→ nextsymb ();
6127→ lex_save_state ();
6128→ nextsymb ();
6129→ }
6130→ lex_restore_state ();
6131→
6132→ switch (s) {
6133→ case S_INCLUDE:
6134→ if (strstr (litbuf, ".co") == (litbuf + (litlen - 3))) {
6135→ mode = LEX_CSOURCE;
6136→ } else {
6137→ mode = LEX_SOURCE; /* #INCLUDE MUST specify the extension */
6138→ }
6139→ break;
6140→ default:
6141→ /* S_SC, S_IMPORT or S_USE, all are descriptor files */
6142→ mode = ((s == S_SC) ? LEX_SC : LEX_LIB);
6143→ current_fe_data->fe_process_filename_fn (litbuf, MAXSTRING_SIZE);
6144→ break;
6145→ }
6146→
6147→ entered = open_file (litbuf, mode, indent);
6148→
6149→ if (!entered) {
6150→ synerr_s (SYN_FILE_OPEN_ERROR, litlocn, litbuf);
6151→ } else {
6152→ if (current_fe_data->fe_information) {
6153→ fprintf (current_fe_data->fe_outfile, "%s \"%s\"\n", tagstring (s), litbuf);
6154→ }
6155→ nextsymb ();
6156→ while (symb == S_NEWLINE) { /* Throw away leading blank lines */
6157→ nextsymb ();
6158→ }
6159→ ignorecomments (0); /* Throw away leading comments */
6160→ }
6161→ /*}}} */
6162→ }
6163→ if (!entered) {
6164→ nextline ();
6165→ }
6166→ return entered;
6167→}
6168→
6169→/*}}}*/
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"file_path": "/home/david/projects/codeassociates/kroc/tools/occ21/fe/lex1.c",
"offset": 1230,
"limit": 60
} 1230→ * if opened ok, otherwise returns FALSE.
1231→ *
1232→ *****************************************************************************/
1233→/*}}}*/
1234→PUBLIC BOOL open_file (const char *const name, const int mode, const int fileindent)
1235→{
1236→ FILE *fptr = NULL;
1237→ BOOL external_ok = FALSE;
1238→
1239→ DEBUG_MSG (("open_file: filename: %s, fileindent: %d\n", name, fileindent));
1240→#if 0
1241→fprintf (stderr, "lex1: open_file: [%s], fileindent = %d\n", name, fileindent);
1242→#endif
1243→ if ((strlen (name) >= MAX_FILENAME_LENGTH) && (mode != LEX_EXTERNAL) && (mode != LEX_DEXTERNAL)) {
1244→ /* bug 331 29/8/90 */
1245→ lexerr_i (LEX_FILE_LENGTH_ERROR, flocn, MAX_FILENAME_LENGTH);
1246→ } else {
1247→ switch (mode) {
1248→ case LEX_SOURCE:
1249→ case LEX_CSOURCE:
1250→ fptr = popen_relative (name, current_fe_data->fe_pathname, NULL, "r", NULL, memalloc, memfree, NULL);
1251→ break;
1252→ case LEX_SC:
1253→ case LEX_LIB:
1254→ case LEX_STDLIB:
1255→ fptr = open_descfile (name, mode);
1256→ break;
1257→ case LEX_EXTERNAL:
1258→ external_ok = init_external (name, 0);
1259→ break;
1260→ case LEX_DEXTERNAL:
1261→ external_ok = init_external (name, 1);
1262→ break;
1263→ }
1264→ }
1265→ if ((fptr != NULL) || external_ok) {
1266→ /*{{{ set up input buffers */
1267→ suspend_file (name, mode, fptr);
1268→ baseindent = fileindent;
1269→ pp_relax = 0; /* reset these */
1270→ pp_iflevel = 0;
1271→ lexmode = mode;
1272→ desc_eof = FALSE;
1273→ preproc_add_define (lookupword ("FILE", 4), 4, PP_VAL_STRING, new_string ((char *)name));
1274→ bufinit ();
1275→ return (TRUE);
1276→ /*}}} */
1277→ } else {
1278→ return (FALSE);
1279→ }
1280→}
1281→/*}}}*/
1282→/*{{{ PRIVATE int readtag (name) */
1283→PRIVATE int readtag (char *const name)
1284→{
1285→ register int i = 0;
1286→ register int temp_ch = ch;
1287→
1288→ if (lexmode != LEX_STDLIB) {
1289→ while (isalnum (temp_ch) || (temp_ch == '.') || ((lexmode == LEX_CSOURCE) ? (temp_ch == '_') : 0)) {
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"pattern": "popen_relative",
"path": "/home/david/projects/codeassociates/kroc/tools/occ21",
"output_mode": "files_with_matches"
}Found 7 files tools/occ21/tcofflib/com.c tools/occ21/include/popen_re.h tools/occ21/include/popen.h tools/occ21/fnilib/popen.c tools/occ21/fe/err1.c tools/occ21/fe/lex1.c tools/occ21/fe/objrdr.c
{
"file_path": "/home/david/projects/codeassociates/kroc/tools/occ21/fnilib/popen.c",
"offset": 100,
"limit": 120
} 100→ , char const * const rf /* relative file name */
101→ , void (* const free_fn)(void *)
102→ /* function to free memory */
103→ )
104→{
105→ fni_handle opened;
106→ fni_style old_style;
107→
108→ /* try host style first */
109→ old_style = fni_set_style (access, host_style);
110→
111→ opened = simple_open (access, fn, md, fl, rf, free_fn);
112→
113→ /* Now try the unix style if we must */
114→ if (opened == NULL)
115→ {
116→ if (fni_set_style (access, unix_style) != unix_style)
117→ opened = simple_open (access, fn, md, fl, rf, free_fn);
118→ }
119→
120→ /* Reset style of file names */
121→ fni_set_style (access, old_style);
122→
123→ return (opened);
124→}
125→
126→/* function popen_relative */
127→
128→/* See popen.h for a specification of this function */
129→
130→FILE * popen_relative (char const * const fn /* name of file to open */
131→ , char const * const ev /* name of env var with path */
132→ , char const * const rn /* name of file relative to */
133→ /* which the open is to be made*/
134→ , char const * const md /* open mode string */
135→ , char const * * const cn /* full file name returned */
136→ , void * (* const malloc_f)(size_t)
137→ /* memory allocation routine */
138→ , void (* const free_f)(void *)
139→ /* memory free routine */
140→ , int * const er /* count of errors in list */
141→ )
142→{
143→ fni_handle opened;
144→ FILE * res;
145→ char const * env_var;
146→ fni_canonical position;
147→ fni_list directories;
148→ fni_access access;
149→ void * (* volatile const malloc_fn)(size_t) = (malloc_f != NULL) ? malloc_f : popen_malloc;
150→ void (* volatile const free_fn)(void *) = (free_f != NULL) ? free_f : free;
151→
152→ errors_found = 0;
153→
154→ if (setjmp(popen_env) == 0)
155→ {
156→ access = fni_open_access(malloc_fn, free_fn);
157→
158→ /* Extract the directory list from the environment variable */
159→ if (ev == NULL)
160→ env_var = NULL;
161→ else
162→ env_var = getenv (ev);
163→ if (env_var == NULL)
164→ directories = NULL;
165→ else
166→ directories = fni_add_dirs_to_list (access, NULL, env_var, mark_error);
167→
168→ /* Now try opening it with this set of environments */
169→ opened = try_open (access, fn, md, directories, rn, free_fn);
170→ if (opened == NULL)
171→ res = NULL;
172→ else
173→ {
174→ /* successful open - return file pointer and full name */
175→ res = fni_system_handle (access, opened);
176→ if (cn != NULL)
177→ {
178→ position = fni_position (access, opened);
179→ *cn = fni_externalise (access, position);
180→ free_fn((void *)position);
181→ }
182→
183→ /* clean up the fni structure */
184→ fni_free (access, opened);
185→ }
186→
187→ /* tidy up the fni structures */
188→ fni_free_list (access, directories);
189→
190→ fni_close_access(&access);
191→
192→ }
193→ else
194→ /* We have longjmp'ed out of a failed memory allocation routine */
195→ res = NULL;
196→
197→ if (er != NULL)
198→ *er = errors_found;
199→
200→ return (res);
201→
202→}
203→
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"pattern": "ISEARCH|isearch|search_path",
"path": "/home/david/projects/codeassociates/kroc/tools/occ21/fnilib/popen.c",
"output_mode": "content"
}Now let me check how the compiler receives the search path - specifically the fe_pathname parameter and open_descfile.
{
"file_path": "/home/david/projects/codeassociates/kroc/tools/occ21/fe/objrdr.c",
"offset": 480,
"limit": 60
} 480→/*{{{ comment */
481→/*****************************************************************************
482→ *
483→ * open_descfile opens the file with name 'name' as a descriptor file
484→ * 'mode' gives the descriptor file mode, either LEX_LIB,
485→ * LEX_STDLIB or LEX_SC.
486→ *
487→ *****************************************************************************/
488→/*}}} */
489→PUBLIC FILE *open_descfile (const char *name, const int mode)
490→{
491→ FILE *fptr = NULL;
492→ (void) mode; /* stop warning */
493→ DEBUG_MSG (("open_descfile: %s\n", name));
494→#if 0 /* sctable no longer used */
495→ if (sctableptr >= MAX_NESTED_SCS)
496→ lexfatal_s (LEX_TOO_MANY_NESTED_SCS, flocn, name);
497→ else
498→#endif
499→ {
500→ fptr = popen_relative (name, current_fe_data->fe_pathname, NULL, "rb", NULL, memalloc, memfree, NULL);
501→ if (fptr != NULL) {
502→#if 0
503→ /* playing around on a Sun; it seems that any non-default
504→ buffering just goes slower.
505→ */
506→ {
507→ const int err = setvbuf (fptr, memalloc (1024), _IOFBF, 1024);
508→ if (err)
509→ abort ();
510→ }
511→#endif
512→ library_eof = FALSE;
513→ {
514→ long int tag = getl_from_tcoff (fptr);
515→ if (LINKED_INPUT_FILE) {
516→ if (tag != LINKED_UNIT_TAG)
517→ lexfatal_s (LEX_TCOFF_NOT_LINKED, flocn, name);
518→ } else {
519→ if (tag != LINKABLE_TAG)
520→ lexfatal_s (LEX_TCOFF_WRONG_FORMAT, flocn, name);
521→ }
522→ throw_record_from_tcoff (fptr);
523→ }
524→ module_level = 0;
525→ in_descriptor = FALSE;
526→ reading_module = FALSE;
527→#ifdef USELIBNAME
528→ libwname = NULL; /* will be set up by LIB_INDEX_START */
529→#endif
530→ file_is_a_library = FALSE;
531→ }
532→ }
533→ return (fptr);
534→}
535→
536→/*}}} */
537→/*{{{ PUBLIC char *readdescriptorline (line) for the lexer */
538→/*{{{ comment */
539→/*****************************************************************************
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"pattern": "fe_pathname|ISEARCH",
"path": "/home/david/projects/codeassociates/kroc/tools/occ21",
"output_mode": "content"
}tools/occ21/include/popen.h:47:#define SYS_SEARCH_PATH "ISEARCH" /* INMOS search path name */ tools/occ21/include/arg.h:329:/* using ISEARCH. If there is nothing else in the argument but @, or */ tools/occ21/include/toolkit.h:177: no = unfold_arg_list (&argc, &argv, switchars, switchopts, "ISEARCH") tools/occ21/include/occamfe.h:88: const char *fe_pathname; /* Environment var. name */ tools/occ21/include/occamfe.h:163: fe_pathname, fe_outfile, fe_errfile, fe_abort_fn, fe_error_occurred tools/occ21/misc/harness.c:167:#define DEFAULT_PATHNAME "ISEARCH" tools/occ21/misc/harness.c:330:/*PUBLIC char pathname[MAX_FILENAME_LENGTH] = "";*/ /* ISEARCH or ZI option */ tools/occ21/misc/harness.c:331:PUBLIC char pathname[MAX_FILENAME_LENGTH] = DEFAULT_PATHNAME; /* ISEARCH or ZI option */ tools/occ21/misc/occompfe.c:117: data->fe_pathname = parms->pathname; tools/occ21/fe/objrdr.c:500: fptr = popen_relative (name, current_fe_data->fe_pathname, NULL, "rb", NULL, memalloc, memfree, NULL); tools/occ21/fe/err1.c:88: FILE *const fptr = popen_relative (fe_fname, current_fe_data->fe_pathname, tools/occ21/fe/lex1.c:1250: fptr = popen_relative (name, current_fe_data->fe_pathname, NULL, "r", NULL, memalloc, memfree, NULL);
Now I have a clear picture. Let me also check to see if #INCLUDE with a .module file is treated differently from regular source includes. Looking back at syn2.c:
Line 6132-6138: For S_INCLUDE (which is #INCLUDE), the mode is LEX_SOURCE (not LEX_LIB). LEX_LIB is only for #USE and #IMPORT. So #INCLUDE "course.module" uses LEX_SOURCE mode.
This means #INCLUDE "course.module" goes through the LEX_SOURCE path, which calls:
Opening the file relative to the current file's directory. Meanwhile, #USE takes the LEX_LIB path and calls open_descfile() to read TCOFF binary format instead. So .module files are actual occam source files that get included textually via #INCLUDE, generated by occbuild's make_module() function. The search path for #INCLUDE defaults to "ISEARCH" and reads from there.
Now I need to trace how occbuild sets up the ISEARCH environment variable or passes the search path to the compiler.
One more important detail - let me check how the ISEARCH path gets set up when the compiler is invoked.
{
"pattern": "ISEARCH",
"path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"output_mode": "content"
}256: old_path = os.getenv("ISEARCH")
261: """Set ISEARCH using search path. This is for the benefit of occ21."""
268: warn("Setting ISEARCH to: ", path)
269: os.putenv("ISEARCH", path){
"pattern": "ISEARCH",
"path": "/home/david/projects/codeassociates/kroc/site_scons",
"output_mode": "content"
}{
"file_path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"offset": 235,
"limit": 50
} 235→ if old_path is not None:
236→ path += ":" + old_path
237→ os.putenv(var, path)
238→
239→def build_default_search_path(tc_search_path):
240→ """Build the default search path, based on environment variables and
241→ the directories the toolchain thinks should be in it."""
242→
243→ global search_path
244→
245→ if in_tree is not None:
246→ # forall and the TVM specials are not included explicitly in
247→ # the search path, so we must add them. Other modules should
248→ # have been listed as dependencies.
249→ search_path += [
250→ in_tree + "/modules/inmoslibs/libsrc/forall",
251→ in_tree + "/tvm/posix",
252→ ]
253→ else:
254→ search_path += tc_search_path
255→
256→ old_path = os.getenv("ISEARCH")
257→ if old_path is not None:
258→ search_path += old_path.split(':')
259→
260→def set_isearch():
261→ """Set ISEARCH using search path. This is for the benefit of occ21."""
262→ if running_windows():
263→ sep = ";"
264→ else:
265→ sep = ":"
266→ path = sep.join(search_path)
267→ if verbose:
268→ warn("Setting ISEARCH to: ", path)
269→ os.putenv("ISEARCH", path)
270→
271→def make_static(library, objs):
272→ """Link a bunch of native objects into a static library."""
273→
274→ run(programs["ar"] + ["rc", library] + objs)
275→ run(programs["ranlib"] + [library])
276→
277→def make_shared(library, objs, extra_opts = []):
278→ """Link a bunch of native objects into a shared library."""
279→
280→ opts = []
281→ if target_os.startswith("darwin"):
282→ opts = ["-dynamiclib", "-read_only_relocs", "suppress"]
283→ elif target_os.startswith("solaris"):
284→ opts = ["-G"]
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
Now let me check how occbuild resolves in-tree module dependencies to build the search path.
{
"pattern": "in.tree.modules|in_tree_modules|search_path.*append|search_path \\+",
"path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"output_mode": "content",
"-B": 2,
"-A": 5
}247- # the search path, so we must add them. Other modules should
248- # have been listed as dependencies.
249: search_path += [
250- in_tree + "/modules/inmoslibs/libsrc/forall",
251- in_tree + "/tvm/posix",
252- ]
253- else:
254: search_path += tc_search_path
255-
256- old_path = os.getenv("ISEARCH")
257- if old_path is not None:
258: search_path += old_path.split(':')
259-
260-def set_isearch():
261- """Set ISEARCH using search path. This is for the benefit of occ21."""
262- if running_windows():
263- sep = ";"
--
489-
490- # Look for all the other modules we need.
491: full_path = search_path + capture(programs["kroc"] + ["--incpath"]).strip().split(":")
492- lib_opts = []
493- for need in need_mods:
494- fn = find_in_path("%s.module" % need, full_path)
495- if fn is None:
496- die("Cannot find module ", need, " in search path ", ":".join(full_path))
--
793-
794- # Look for all the other modules we need.
795: full_path = search_path + ["."] + capture(programs["tock"] + ["--module-path"]).strip().split(":")
796- lib_opts = []
797- resolved_mods = []
798- for need in need_mods:
799- fn = find_in_path("%s.module" % need, full_path)
800- if fn is None:
--
1069- toolchain = toolchains[a]()
1070- elif o == "--search":
1071: search_path.append(a)
1072- elif o == "--occ21-opts":
1073- occ21_opts += split_options(a)
1074- elif o == "-D":
1075- occ21_opts += ["-DEF", a]
1076- tock_opts += ["-D", a]
--
1147- added_paths.append(os.path.normpath(os.path.join(sd, rel)))
1148-
1149: search_path += added_paths
1150-
1151- # Tidy up the search path, now we've added everything to it.
1152- tidy_search_path = []
1153- for dir in search_path:
1154- # Make the paths absolute, and normalise them.
--
1157- # Remove duplicate paths.
1158- if not dir in tidy_search_path:
1159: tidy_search_path.append(dir)
1160- search_path = tidy_search_path
1161-
1162- set_isearch()
1163- for dir in search_path:
1164- programs["cc"] += ["-L" + dir]{
"file_path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"offset": 1120,
"limit": 50
} 1120→
1121→ #{{{ construct and export search_path
1122→ build_default_search_path(toolchain.search_path())
1123→
1124→ if mode in ("object", "program"):
1125→ # If we're building a source file in another directory
1126→ # (e.g. ../../libsrc/foo.occ), then we need to add search paths
1127→ # relative to that directory too. This handles the case when
1128→ # we're building with a separate object directory (even for
1129→ # modules in the whole KRoC tree).
1130→ here = os.getcwd()
1131→ for source in [target] + args:
1132→ if source is None or source == "":
1133→ continue
1134→
1135→ sd = os.path.dirname(source)
1136→ if sd != ".":
1137→ added_paths = [sd]
1138→
1139→ for dir in search_path:
1140→ # Work out the relative path from the
1141→ # current directory to the directory in
1142→ # the path.
1143→ rel = make_relative_path(here, dir)
1144→
1145→ # Follow the same relative path from
1146→ # the source directory.
1147→ added_paths.append(os.path.normpath(os.path.join(sd, rel)))
1148→
1149→ search_path += added_paths
1150→
1151→ # Tidy up the search path, now we've added everything to it.
1152→ tidy_search_path = []
1153→ for dir in search_path:
1154→ # Make the paths absolute, and normalise them.
1155→ dir = os.path.normpath(os.path.abspath(dir))
1156→
1157→ # Remove duplicate paths.
1158→ if not dir in tidy_search_path:
1159→ tidy_search_path.append(dir)
1160→ search_path = tidy_search_path
1161→
1162→ set_isearch()
1163→ for dir in search_path:
1164→ programs["cc"] += ["-L" + dir]
1165→ programs["kroc"] += ["-I" + dir, "-L" + dir]
1166→ #}}}
1167→
1168→ #{{{ construct filename patterns
1169→ global patterns, build_shared
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
Now let me check the --need flag in occbuild, which is how dependencies between modules are declared and how the search path gets populated with module directories.
{
"pattern": "\"--need\"|need_mods|in-tree-modules",
"path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"output_mode": "content",
"-B": 2,
"-A": 5
}484- make_module(base, custom)
485-
486: need_mods = needs[:]
487- if std_libs:
488: need_mods.append("forall")
489-
490- # Look for all the other modules we need.
491- full_path = search_path + capture(programs["kroc"] + ["--incpath"]).strip().split(":")
492- lib_opts = []
493: for need in need_mods:
494- fn = find_in_path("%s.module" % need, full_path)
495- if fn is None:
496- die("Cannot find module ", need, " in search path ", ":".join(full_path))
497-
498- f = open(fn)
--
776-
777- def object(self, source):
778: need_mods = needs[:]
779- impl = []
780- if std_libs:
781: need_mods.append("forall")
782- impl.append("-u")
783- impl.append("forall.module")
784: run(self.tock_cmd(["-c"]) + impl + [source]) #["-u" + m for m in need_mods] +
785- (base, ext) = split_ext(source)
786-#TODO we should only make the stub if compilation succeeded and put out a .tock.o file
787- make_stub(base + ".tce")
788-
789- def library(self, occam_objs, native_objs, base):
790: need_mods = needs[:]
791- if std_libs:
792: need_mods.append("forall")
793-
794- # Look for all the other modules we need.
795- full_path = search_path + ["."] + capture(programs["tock"] + ["--module-path"]).strip().split(":")
796- lib_opts = []
797- resolved_mods = []
798: for need in need_mods:
799- fn = find_in_path("%s.module" % need, full_path)
800- if fn is None:
801- die("Cannot find module ", need, " in search path ", ":".join(full_path))
802- resolved_mods += [fn]
803-
--
861-
862- def program(self, occam_objs, native_objs, source, output):
863: need_mods = needs[:]
864- impl = []
865- if std_libs:
866: need_mods.append("forall")
867- impl.append("-u")
868- impl.append("forall.module")
869-
870- run(self.tock_cmd(["--linker-flags", " ".join(native_objs)] + impl + ["-o", output]) + [source])
871-#" ".join([split_ext(o)[0] + ".tock.o" for o in occam_objs] +
--
1077- elif o in ("-l", "-L", "-r", "-R", "-W", "-m", "-p"):
1078- linker_opts.append("%s%s" % (o, a))
1079: elif o == "--need":
1080- needs.append(a)
1081- elif o == "--include":
1082- includes_before.append(a)
1083- elif o == "--include-after":
1084- includes_after.append(a){
"file_path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"offset": 1090,
"limit": 35
} 1090→ prefix = a
1091→ elif o == "--destdir":
1092→ destdir = a
1093→ elif o == "--no-std-libs":
1094→ std_libs = False
1095→ else:
1096→ for prog in programs.keys():
1097→ if o == "--" + prog:
1098→ programs[prog][0] = a
1099→ elif o == "--" + prog + "-opts":
1100→ programs[prog] += split_options(a)
1101→
1102→ if mode is None:
1103→ usage(sys.stderr)
1104→ sys.exit(1)
1105→ #}}}
1106→
1107→ if toolchain is None:
1108→ toolchain = toolchains["@OCCBUILD_TOOLCHAIN@"]()
1109→
1110→ #{{{ adjust program locations for --in-tree
1111→ programs["tvm_installed"] = programs["tvm"]
1112→ if in_tree is not None:
1113→ programs["kroc"] = [in_tree + "/tools/kroc/kroc", "--in-tree", in_tree] + programs["kroc"][1:]
1114→ programs["ilibr"] = [in_tree + "/tools/ilibr/ilibr"] + programs["ilibr"][1:]
1115→ programs["occ21"] = [in_tree + "/tools/occ21/occ21"] + programs["occ21"][1:]
1116→ programs["plinker.pl"] = [in_tree + "/tools/plinker/plinker.pl"] + programs["plinker.pl"][1:]
1117→ programs["tce-dump.pl"] = [in_tree + "/tools/plinker/tce-dump.pl"] + programs["tce-dump.pl"][1:]
1118→ programs["tvm"] = [in_tree + "/tvm/posix/tvm"] + programs["tvm"][1:]
1119→ #}}}
1120→
1121→ #{{{ construct and export search_path
1122→ build_default_search_path(toolchain.search_path())
1123→
1124→ if mode in ("object", "program"):
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"file_path": "/home/david/projects/codeassociates/kroc/site_scons/site_tools/occbuild.py"
} 1→# See: http://www.scons.org/wiki/UsingCodeGenerators
2→from SCons.Builder import Builder
3→from SCons.Action import Action
4→from SCons.Scanner import Scanner, FindPathDirs
5→from SCons.Environment import Environment
6→from SCons.Util import AddMethod, CLVar
7→import os
8→import pideps
9→
10→# Monkeypatching
11→pideps.sourceprefix = ""
12→pideps.die = pideps.warn
13→def find_dep(fn, dep, incpath):
14→ for dir in [os.path.dirname(fn)] + incpath:
15→ dpath = os.path.normpath(os.path.join(dir, dep))
16→ if os.access(dpath, os.F_OK):
17→ return dpath
18→ return os.path.join(os.path.dirname(fn), dep)
19→pideps.find_dep = find_dep
20→
21→
22→# TODO: This generally didn't work out as well as I'd hoped, but it it still
23→# better than typing in dependencies by hand for now.
24→# This can be looked at later.
25→def pideps_scan(node, env, path):
26→ # FIXME: occbuild sets OCCBUILD.TVM and presumably OCCBUILD.KROC, add those
27→ # here automatically? Evaluate and extract -D out the $OCCBUILDCOMSTR to see
28→ # what to add here
29→ defines = {}
30→ binaries = []
31→ deps = {}
32→ path = [os.path.abspath(d.path) for d in path]
33→ pideps.parse(node.path, deps, path, binaries, defines)
34→ deplines = deps.keys()
35→ #deplines.sort()
36→ deplines = [l.split(' ')[1].split(' ') for l in deplines]
37→ deplines = reduce(lambda x, y: x + y, deplines, [])
38→ deplines = [os.path.abspath(x) for x in deplines]
39→ print deplines
40→ return deplines
41→
42→# This emitter will be used later by a Builder, and set an explicit dependency
43→# on the argument passed in as 'dependency', most probably occbuild
44→def occbuild_depend_emitter(target, source, env, dependency):
45→ env.Depends(target, dependency)
46→ return (target, source)
47→
48→def occbuild_program_emitter(target, source, env):
49→ # FIXME: For kroc compile no .tbc file will be generated
50→ target_name = str(target[0])
51→ if target[0].suffix:
52→ tbc = target_name.replace(target[0].suffix, '.tbc')
53→ else:
54→ tbc = target_name + '.tbc'
55→ target = target + [tbc]
56→ return (target, source)
57→
58→def occbuild_library_emitter(target, source, env):
59→ # FIXME: for kroc compile, no precomp will be generated
60→ target_name = str(target[0])
61→ precomp = target_name.replace(target[0].suffix, '.precomp')
62→ module = target_name.replace(target[0].suffix, '.module')
63→ if len(source) != 0:
64→ # No precomp is generated if there are no sources, ie a 'meta' library.
65→ target.append(precomp)
66→ target.append(module)
67→ return (target, source)
68→
69→def generate(env, **kw):
70→ occbuild = kw.get('occbuild', None)
71→ if occbuild:
72→ occbuild_path = occbuild[0].abspath
73→ depend_emitter = lambda target, source, env: occbuild_depend_emitter(target, source, env, occbuild)
74→ else:
75→ occbuild_path = 'occbuild'
76→ depend_emitter = None
77→ pideps_scanner = Scanner(function = pideps_scan, skeys = ['.occ'], path_function = FindPathDirs('INCPATH'))
78→ tce_bld = Builder(action = Action('$OCCBUILDCOM', '$OCCBUILDCOMSTR'),
79→ emitter = depend_emitter,
80→ suffix = '.tce',
81→ src_suffix = '.occ')
82→ # FIXME: The source scanner does not work well enough yet :/
83→ #source_scanner = pideps_scanner)
84→ lib_bld = Builder(action = Action('$OCCBUILDLIBRARYCOM', '$OCCBUILDLIBRARYCOMSTR'),
85→ emitter = [depend_emitter, occbuild_library_emitter],
86→ suffix = '.lib',
87→ src_suffix = '.tce',
88→ src_builder = [tce_bld])
89→ prog_bld = Builder(action = Action('$OCCBUILDPROGRAMCOM', '$OCCBUILDPROGRAMCOMSTR'),
90→ emitter = [depend_emitter, occbuild_program_emitter],
91→ suffix='$PROGSUFFIX',
92→ src_suffix = ['.occ', '.tce'],
93→ # FIXME: If I leave the sourcebuilder in, scons seems to
94→ # want to turn my .occ extensions when I have a mixed
95→ # .occ, .tce source list into .tce using the builder
96→ )#src_builder = [tce_bld])
97→ tbc_headr_bld = Builder(action = Action('$TBCHEADERCOM', '$TBCHEADERCOMSTR'),
98→ emitter = [depend_emitter],
99→ suffix='.h',
100→ src_suffix = ['.occ', '.tce'],
101→ src_bulider = [tce_bld])
102→ # Add the new Builder to the list of builders
103→ # Use of $( $) causes bracketed flags not trigger rebuild when changed
104→ env['BUILDERS']['OccamObject'] = tce_bld
105→ env['OCCBUILDCOM'] = '$OCCBUILD $_OCCBUILD_TOOLCHAIN $_OCCBUILD_SEARCH_DIRS $OCCBUILDFLAGS --object $SOURCES'
106→ env['BUILDERS']['OccamLibrary'] = lib_bld
107→ env['OCCBUILDLIBRARYCOM'] = '$OCCBUILD $_OCCBUILD_TOOLCHAIN $_OCCBUILD_SEARCH_DIRS $OCCBUILDFLAGS --library $TARGET $SOURCES'
108→ env['BUILDERS']['OccamProgram'] = prog_bld
109→ env['OCCBUILDPROGRAMCOM'] = '$OCCBUILD $_OCCBUILD_TOOLCHAIN $_OCCBUILD_SEARCH_DIRS $OCCBUILDFLAGS --program $SOURCES'
110→ env['BUILDERS']['OccamBytecodeHeader'] = tbc_headr_bld
111→ env['TBCHEADERCOM'] = '$SKROC $_SKROC_SEARCH_DIRS $SKROCFLAGS --c -f $TARGET $SOURCES'
112→ env['OCCBUILD'] = occbuild_path
113→ env['_OCCBUILD_SEARCH_DIRS'] = '$( ${_concat(OCCBUILD_SEARCH_PFX, INCPATH, "", __env__, RDirs, TARGET, SOURCE)} $)'
114→ env['_SKROC_SEARCH_DIRS'] = '$( ${_concat(SKROC_SEARCH_PFX, INCPATH, "", __env__, RDirs, TARGET, SOURCE)} $)'
115→ env['OCCBUILD_SEARCH_PFX'] = '--search '
116→ env['SKROC_SEARCH_PFX'] = '-L '
117→ env['OCCBUILD_TOOLCHAIN'] = None
118→ env['_OCCBUILD_TOOLCHAIN'] = '${(OCCBUILD_TOOLCHAIN and "--toolchain $OCCBUILD_TOOLCHAIN" or "")}'
119→ def OccLibDepend(self, node, lib_name):
120→ if not isinstance(lib_name, list): list(lib_name)
121→ for lib in lib_name:
122→ self.Depends(node, self['OCCLIBS'][lib]['dep'])
123→ if 'inc' in self['OCCLIBS'][lib]:
124→ for n in node:
125→ n.env.AppendUnique(INCPATH=self['OCCLIBS'][lib]['inc'])
126→ env.AddMethod(OccLibDepend)
127→ env['OCCLIBS'] = dict()
128→ env['INCPATH'] = CLVar('')
129→ env['OCCBUILDFLAGS'] = CLVar('')
130→
131→
132→
133→def exists(env):
134→ return 1
135→
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
Now I have the complete picture. Here's the explanation:
#INCLUDE "course.module" WorksThe include mechanism has three layers: the C compiler's file resolution, the occbuild Python tool that generates .module files and manages search paths, and the build system (SCons/autoconf) that orchestrates everything.
occ21)When the compiler parses #INCLUDE "course.module" in syn2.c:6086 (rfile()), it treats it as a plain source include (LEX_SOURCE mode). It calls open_file() at lex1.c:1234, which calls:
fptr = popen_relative(name, current_fe_data->fe_pathname, NULL, "r", ...);
The second argument, fe_pathname, defaults to the string "ISEARCH" (misc/harness.c:331). This tells popen_relative() (in fnilib/popen.c:130) to read the ISEARCH environment variable, split it into a colon-separated list of directories, and search each one for the file. The compiler itself knows nothing about modules, build systems, or Python scripts -- it just searches directories listed in ISEARCH.
occbuild -- The Module BuilderThe real intelligence lives in tools/kroc/occbuild.in, a Python script. It does two critical things:
A) Generates .module files. The make_module() function (occbuild.in:290) creates course.module as a real occam source file on disk. For the course library, the generated file looks roughly like:
-- Include file for "course" module
#IF NOT (DEFINED (COURSE.MODULE))
#DEFINE COURSE.MODULE
#INCLUDE "forall.module"
#INCLUDE "consts.inc"
... toolchain-specific PRAGMA directives ...
#ENDIF
The content is controlled by --need (adds #INCLUDE "X.module" for dependencies), --include (adds custom #INCLUDE lines), and a toolchain-specific callback that adds #PRAGMA directives to link the compiled library.
B) Sets the ISEARCH environment variable. Before invoking occ21, occbuild builds a search path from multiple sources and calls set_isearch() (occbuild.in:260) to export it:
in-tree-modules file (see Layer 3)--search flags -- passed by the build system (occbuild.in:1071)ISEARCH -- any user-set value is appended (occbuild.in:256)occbuild.in:254)SCons (site_scons/site_tools/occbuild.py): Each INCPATH directory in the SConscript environment is converted to a --search flag for occbuild (line 113-115). So when libsrc/SConscript sets INCPATH='.', the libsrc/ directory ends up in the search path.
Autoconf (m4/occbuild.m4): The OCCAM_PROVIDE macro (occbuild.m4:218) registers modules in a file called $KROC_BUILD_ROOT/in-tree-modules. Each line maps a module name to its directory:
course.module /path/to/modules/course/libsrc
shared_screen.module /path/to/modules/course/libsrc course.module
When occbuild runs with --in-tree, it reads this file to discover where each dependency's .module file lives and adds those directories to the search path.
hello_world.occ
│ #INCLUDE "course.module"
▼
Build system (SCons) invokes:
occbuild --search /path/to/libsrc --need course --object hello_world.occ
│
▼
occbuild:
1. Builds search_path from --search flags, in-tree-modules, ISEARCH env
2. Sets ISEARCH="/path/to/libsrc:/path/to/forall:..."
3. Invokes occ21 (the C compiler)
│
▼
occ21:
1. Parses #INCLUDE "course.module"
2. Calls popen_relative("course.module", "ISEARCH", ...)
3. Reads ISEARCH env var, searches each directory
4. Finds and opens /path/to/libsrc/course.module
5. Parses it as occam source (it's a generated .occ-like file)
Your hunch was partly right -- Python scripts are involved, but they don't dynamically discover source files at include-resolution time. Instead, occbuild is invoked at build time to (a) generate the .module file from the library's declared dependencies, and (b) set up the ISEARCH environment so the C compiler can find it. The SConscript files orchestrate which sources go into each library and what flags get passed to occbuild. The compiler itself is just doing a simple path-based file search.
The user wants to see what a generated course.module file actually looks like. Let me trace through the make_module() function more carefully to reconstruct what it would generate for the course library. I need to understand what needs, includes_before, includes_after, and the custom callback would contain for the course library build.
Let me look at how the course library is built - specifically the KRoCToolchain's library() method which calls make_module().
Let me trace through the exact code path that generates it. I need to see the toolchain's library() method and what it passes to make_module().
{
"pattern": "def library",
"path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"output_mode": "content"
}411: def library(self, occam_objs, native_objs, base): 472: def library(self, occam_objs, native_objs, base): 644: def library(self, occam_objs, native_objs, base): 789: def library(self, occam_objs, native_objs, base):
{
"file_path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"offset": 411,
"limit": 100
} 411→ def library(self, occam_objs, native_objs, base):
412→ """Link occam object files (.tce) and native object files (.o)
413→ into a library, giving base.lib."""
414→ pass
415→
416→ def program(self, occam_objs, native_objs, source, output):
417→ """Compile source, and link it with occam object files (.tce)
418→ and native object files (.o) into an executable, giving
419→ output."""
420→ pass
421→
422→ def run(self, program, args):
423→ """Run executable program. (This may involve setting up
424→ environment variables or using an interpreter.)"""
425→ pass
426→
427→ def install(self, file):
428→ """Install file (and its associated files) into an appropriate
429→ directory based on its extension -- .lib, .inc, .h, or nothing
430→ for an executable. Executables should be installed into a
431→ directory that will be in the user's path."""
432→ pass
433→
434→ def install_examples(self, package, file):
435→ """Install a program executable that is an example for a
436→ package; this will typically not be on the user's path."""
437→ pass
438→
439→ def clean(self, file):
440→ """Delete file along with any other files that were generated
441→ during its compilation."""
442→ pass
443→
444→ def cflags(self):
445→ """Print to stdout the CFLAGS necessary when compiling native
446→ objects for this toolchain. This should at least include
447→ -DOCCBUILD_nameoftoolchain."""
448→ pass
449→
450→ def search_path(self):
451→ """Return a list of directories that should be included in
452→ search_path when compiling outside the KRoC tree."""
453→ pass
454→
455→ def update_patterns(self, patterns):
456→ """Adjust the patterns dict -- for example, changing library
457→ prefixes to something other than the default."""
458→ pass
459→#}}}
460→#{{{ class KRoCToolchain
461→class KRoCToolchain(Toolchain):
462→ def kroc_cmd(self):
463→ cmd = programs["kroc"][:]
464→ cmd += ["--oc-opts", " ".join(occ21_opts + ["-DEF", "OCCBUILD.KROC"])]
465→ if static_link:
466→ cmd += ["--cc-opts", "-static"]
467→ return cmd
468→
469→ def object(self, source):
470→ run(self.kroc_cmd() + ["-c", source])
471→
472→ def library(self, occam_objs, native_objs, base):
473→ def custom(f):
474→ if occam_objs != []:
475→ f.write('#USE "%s.lib"\n' % base)
476→ if occam_objs != [] or native_objs != []:
477→ f.write('#PRAGMA COMMENT ".USELIB %s"\n'
478→ % base)
479→ f.write('-- occbuild:has-native-library\n')
480→ if linker_opts != []:
481→ f.write('#PRAGMA COMMENT ".LDFLAGS %s"\n'
482→ % ' '.join(linker_opts))
483→ f.write('\n')
484→ make_module(base, custom)
485→
486→ need_mods = needs[:]
487→ if std_libs:
488→ need_mods.append("forall")
489→
490→ # Look for all the other modules we need.
491→ full_path = search_path + capture(programs["kroc"] + ["--incpath"]).strip().split(":")
492→ lib_opts = []
493→ for need in need_mods:
494→ fn = find_in_path("%s.module" % need, full_path)
495→ if fn is None:
496→ die("Cannot find module ", need, " in search path ", ":".join(full_path))
497→
498→ f = open(fn)
499→ data = f.read()
500→ f.close()
501→
502→ # If this module uses a native library, we must link
503→ # against it when building a shared library.
504→ if data.find('occbuild:has-native-library') != -1:
505→ lib_opts.append(patterns["link"] % need)
506→
507→ if std_libs:
508→ lib_opts.append(patterns["rtlink"] % "krocif")
509→ lib_opts.append(patterns["rtlink"] % "ccsp")
510→
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
Now I can reconstruct exactly what gets generated. Let me also check what needs, includes_before, and std_libs would be for the course library build. Recall from the SConscript:
{
"file_path": "/home/david/projects/codeassociates/kroc/modules/course/libsrc/SConscript",
"offset": 22,
"limit": 10
} 22→course_lib = local.OccamLibrary(
23→ 'course.lib',
24→ course_lib_srcs,
25→ INCPATH='.',
26→ OCCBUILDFLAGS='--include consts.inc')
27→
28→def mk_shared_screen(lib_name):
29→ return local.OccamLibrary(
30→ lib_name,
31→ shared_screen_lib_objs,
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
{
"pattern": "std_libs|includes_before|includes_after|^needs\"",
"path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"output_mode": "content",
"-B": 0,
"-A": 0
}30:std_libs = True
39:includes_before = []
40:includes_after = []
312: do_list('#INCLUDE "%s"\n', includes_before)
314: do_list('#INCLUDE "%s"\n', includes_after)
487: if std_libs:
507: if std_libs:
679: if std_libs:
780: if std_libs:
791: if std_libs:
865: if std_libs:
1045: global std_libs, verbose, in_tree, search_path
1047: global includes_before, includes_after, static_link
1082: includes_before.append(a)
1084: includes_after.append(a)
1094: std_libs = False{
"file_path": "/home/david/projects/codeassociates/kroc/tools/kroc/occbuild.in",
"offset": 25,
"limit": 20
} 25→#{{{ globals
26→VERSION = "$Revision$"[11:-2]
27→CONTACT = "kroc-bugs@kent.ac.uk"
28→autogen_message = "Automatically generated by occbuild %s; do not edit" % VERSION
29→
30→std_libs = True
31→verbose = False
32→in_tree = None
33→programs = {}
34→search_path = []
35→occ21_opts = []
36→tock_opts = []
37→linker_opts = []
38→needs = []
39→includes_before = []
40→includes_after = []
41→static_link = False
42→prefix = None
43→destdir = ""
44→target_alias = "@target_alias@"
<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>
Now I have all the pieces. For the course library build, the SCons command expands to something like:
occbuild --search . --include consts.inc --library course.lib utils.tce string.tce ...
So:
- needs = [] (no --need flags)
- includes_before = ["consts.inc"] (from --include consts.inc)
- includes_after = []
- std_libs = True
- base = "course"
- occam_objs is non-empty (the .tce files)
- native_objs is empty (no native C objects)
- linker_opts = []
Walking through make_module("course", custom) at occbuild.in:290:
guard = "COURSE.MODULE" # (line 300: ofn="course.module", replace _ with ., upper())
needs = [] # so do_list on line 311 outputs nothing
includes_before = ["consts.inc"] # line 312 outputs: #INCLUDE "consts.inc"
custom(f): # line 313 — the KRoCToolchain closure at line 473:
occam_objs != [] → True → writes: #USE "course.lib"
occam_objs != [] → True → writes: #PRAGMA COMMENT ".USELIB course"
→ writes: -- occbuild:has-native-library
linker_opts == [] → skips LDFLAGS line
includes_after = [] # line 314 outputs nothing
So the generated course.module file would be:
-- Include file for "course" module
-- Automatically generated by occbuild ...; do not edit
#IF NOT (DEFINED (COURSE.MODULE))
#DEFINE COURSE.MODULE
#INCLUDE "consts.inc"
#USE "course.lib"
#PRAGMA COMMENT ".USELIB course"
-- occbuild:has-native-library
#ENDIF
That's it. It's a small generated occam source fragment that:
1. Has an include guard (#IF NOT DEFINED / #DEFINE) to prevent double-inclusion
2. Includes consts.inc (the constants like NULL, BELL, etc. we saw earlier)
3. #USE "course.lib" — tells the compiler to read the compiled library's type descriptors (this is the TCOFF binary path via open_descfile)
4. #PRAGMA COMMENT ".USELIB course" — embeds a linker directive so the final link step knows to pull in the course native library
5. -- occbuild:has-native-library — a comment marker that occbuild itself scans for when building downstream libraries that depend on course (to know it needs -lcourse linker flags)
Thank you and apologies I hit enter accidentally above. It turns out there are no generated .module files in the git repository so they must be built at install time, or sometime between installation and build time.
The user is confirming that .module files aren't checked into git - they're generated during the build process. This makes sense given what we've seen: make_module() in occbuild.in generates them, and the SCons occbuild_library_emitter (in occbuild.py:58) adds the .module file as a build target alongside the .lib file. So they're generated as part of the library build step, before any dependent source files are compiled.
Right, they're generated at build time — specifically as part of the library build step. You can see this in site_scons/site_tools/occbuild.py:58-67: the occbuild_library_emitter adds the .module file as a build target alongside the .lib:
def occbuild_library_emitter(target, source, env):
target_name = str(target[0])
precomp = target_name.replace(target[0].suffix, '.precomp')
module = target_name.replace(target[0].suffix, '.module')
...
target.append(module)
So when SCons builds course.lib, it also produces course.module as an output artifact in the same step. The dependency graph ensures this happens before any downstream .occ files that #INCLUDE "course.module" are compiled.
Thank you. If I were working on making another occam compiler and didn't want yet to implement all of this, do you think it would be feasible for me to run occbuild once to generate the .module and .lib files, then copy them into my project and have my compiler just to simple path include? Woud that work?
The user is asking whether they could pre-generate the .module and .lib files using occbuild, then use them with their own compiler via simple path-based inclusion.
Let me think about what the .module files contain and what the compiler needs to do with them:
.module files are plain occam source text - #IF, #DEFINE, #INCLUDE, #USE, #PRAGMA directives. A simple path-based #INCLUDE would work for reading them.
But the .module files contain #USE "course.lib" — this is where it gets tricky. #USE in occ21 triggers open_descfile() which reads a TCOFF binary format file.
The .lib file is a binary containing type descriptors and procedure signatures, so any new compiler would need to understand the TCOFF format to extract that type information. The #PRAGMA COMMENT directives are just linker hints that can pass through unchanged, but the .module files also pull in other dependencies like plain occam source files and potentially other .module files, meaning you'd need to resolve the entire dependency tree.
So it's partially possible — you could use the .module files as-is with path-based inclusion, but you'd still need to handle TCOFF parsing for the .lib files and manage all the transitive dependencies. The consts.inc file would work fine since it's pure occam source, but the real challenge is that the .module files reference .lib files through #USE directives, which requires parsing TCOFF binary format—something that's not trivial to implement. The user could potentially work around this by either implementing #USE support or manually replacing those directives with the actual occam declarations they need.
Mostly yes, but with one significant complication. The textual #INCLUDE chain would work fine — .module files are plain occam source, and things like consts.inc are just VAL declarations. Your compiler would need to handle the preprocessor directives (#IF NOT (DEFINED ...), #DEFINE, #ENDIF) but those are straightforward.
The problem is the #USE "course.lib" line inside the generated .module file. That's not a textual include — in occ21 it triggers open_descfile() which reads course.lib as a TCOFF binary containing type descriptors, procedure signatures, channel protocol definitions, etc. That's how the compiler learns the types of everything exported by the library without re-parsing all the source. Implementing a TCOFF reader is a non-trivial piece of work.
A couple of workarounds you could consider:
Replace #USE with source-level declarations. Instead of using the generated .module files as-is, create your own hand-written .module files that #INCLUDE the actual occam source headers (like consts.inc) and contain the PROC / PROTOCOL / CHAN TYPE declarations directly as occam source. Essentially flatten everything into text your compiler can parse.
Start with self-contained programs. For something like hello_world.occ, you could inline the few declarations it actually needs (the VAL BYTE constants, the out.string signature) and skip the module system entirely while you get the core language working.
The .lib files would still be needed later at link time regardless, but that's a separate concern from compilation.
Ok thanks, now I see. I will try the textural include idea, as compilers are now so fast that the performance benefit of the binary precompile approach is probably not necessary. Your idea of inlining the declarations required for simple programs is also something I considered but at present my goal is to compile actual Occam programs from long ago. Those all have #use "course.module" and I'd like to be able to compile them as-is.
The user is saying they want to compile existing occam programs unmodified, which contain #INCLUDE "course.module". They want to take the textual include approach - replacing the #USE "course.lib" inside the .module files with actual occam source declarations.
So their plan would be:
1. Create their own course.module file that, instead of #USE "course.lib", directly #INCLUDEs the relevant occam source files that declare the procedures/types
2. Their compiler just needs to handle #INCLUDE as textual inclusion
This is a reasonable approach. The source files in modules/course/libsrc/ contain the actual procedure implementations. They'd need to extract just the declarations (PROC signatures, VAL constants, etc.) or include the full source files.
I'm realizing that since #INCLUDE does textual substitution, they could include the actual .occ source files directly, though that would pull in full implementations rather than just interfaces. The tradeoff is that each program would get complete copies of the implementations compiled in, which isn't ideal but would at least get things working. Let me double-check what the original programs actually use - they specified #INCLUDE "course.module", so I need to verify whether that's the right directive. _world.occ has #INCLUDE "course.module", and the .module file in turn has #USE "course.lib". So the user's compiler would need to handle #INCLUDE (textual) which brings in the .module file, which contains #USE (binary).
The user's idea is to create a version of course.module that replaces the #USE "course.lib" with textual #INCLUDE of the actual source declarations. This way their compiler only needs #INCLUDE (textual) support.
This is a sound approach. They just need to be careful that they're getting the right declarations. The source files like utils.occ, string.occ, etc. contain full PROC bodies. In occam, if you #INCLUDE a file with PROC bodies, you get those compiled into your program—that's different from #USE which just imports the interface.
For a working compiler, they could either extract just the declarations and signatures, or include the full source so the library code gets compiled into every program. The second option is simpler and probably fine for getting things working, even if it's less efficient.
The user seems to have a clear grasp of what they're doing, so I should keep my response brief and just acknowledge their approach while noting the distinction between declarations and full implementations in case it matters for their compiler design.
That makes sense. So you'd create your own course.module that replaces the #USE "course.lib" line with #INCLUDEs of the actual occam source files — keeping the existing programs completely unmodified.
One thing to be aware of: #USE imports only the interface (procedure signatures, type declarations) without the implementation bodies, whereas #INCLUDE-ing the full .occ source files would pull in the complete procedure bodies. Depending on how your compiler handles separate compilation and linking, you might either:
.inc file and include that, then link the library separatelyEither way, it should let you compile those old programs with #INCLUDE "course.module" unchanged.