NameDateSize

..16-Mar-201612 KiB

.classpath26-Aug-20151.2 KiB

.gitattributes02-May-2013329

.gitignore27-Apr-2015801

.project02-May-2013367

.travis.yml17-Apr-2015725

build.xml30-Nov-201543.9 KiB

CHANGES.txt29-Apr-201435.9 KiB

checkstyle/02-May-20134 KiB

codestyle-jel.eclipse.xml02-May-201328.9 KiB

debian/04-Feb-20144 KiB

doc/09-Apr-20144 KiB

egrok/02-Oct-20144 KiB

ext/02-May-20134 KiB

jrcs/29-Apr-20144 KiB

lib/27-Apr-20154 KiB

LICENSE.txt02-May-201318.6 KiB

logging.properties11-Dec-20152.7 KiB

nbproject/15-Dec-20154 KiB

NOTICE.txt02-May-20131.3 KiB

OpenGrok20-Nov-201532.3 KiB

opengrok-indexer/07-Apr-20154 KiB

opengrok-web/29-Apr-20144 KiB

opengrok-web-nbproject/02-May-20134 KiB

oracle.eclipse.tools.webtier.ui.prefs02-May-2013156

org.eclipse.jdt.core.prefs02-May-2013618

paths.tsv02-May-2013113

platform/02-May-20134 KiB

pom.xml15-Dec-20159.7 KiB

README.txt12-Oct-201534 KiB

sonar-project.properties07-Aug-2015699

src/02-May-20134 KiB

test/02-May-20134 KiB

testdata/02-May-20134 KiB

tools/02-May-20134 KiB

userlib-jasper.eclipse02-May-2013828

web/11-Dec-20154 KiB

README.txt

1#
2# Copyright (c) 2006, 2015, Oracle and/or its affiliates. All rights reserved.
3#
4
5
6OpenGrok - a wicked fast source browser
7---------------------------------------
8
91.  Introduction
102.  Requirements
113.  Usage
124.  OpenGrok install
135.  OpenGrok setup
146.  Optional Command Line Interface Usage
157.  Change web application properties or name
168.  OpenGrok systray
179.  Information for developers
1810. Tuning OpenGrok for large code bases
1911. Authors
2012. Contact us
21
22
231. Introduction
24---------------
25
26OpenGrok is a fast and usable source code search and cross reference
27engine, written in Java. It helps you search, cross-reference and navigate
28your source tree. It can understand various program file formats and
29version control histories like Mercurial, Git, SCCS, RCS, CVS, Subversion,
30Teamware, ClearCase, Perforce, Monotone and Bazaar.
31
32Offical page of the project is on:
33
34  http://opengrok.github.com/OpenGrok/
35
362. Requirements
37---------------
38
39    * Latest Java (At least 1.8)
40      http://www.oracle.com/technetwork/java/
41    * A servlet container like Tomcat (8.x or later)
42      supporting Servlet 2.5 and JSP 2.1
43      http://tomcat.apache.org/
44    * Exuberant Ctags or Universal Ctags
45      http://ctags.sourceforge.net/
46      http://github.io/uctags
47    * Source Code Management installation
48      depending on type of repositories indexed
49    * If you want to build OpenGrok:
50      - Ant (1.9.4 and later)
51        http://ant.apache.org/
52      - JFlex
53        http://www.jflex.de/
54      - Netbeans (optional, at least 8.0, will need Ant 1.9.4)
55        http://netbeans.org/
56
573. Usage
58--------
59
60OpenGrok usually runs in servlet container (e.g. Tomcat).
61
62SRC_ROOT environment variable refers to the directory containing your source
63tree. OpenGrok analyzes the source tree and builds a search index along with
64cross-referenced hypertext versions of the source files. These generated
65data files will be stored in directory referred to with environment variable
66called DATA_ROOT.
67
683.1 Projects
69------------
70
71OpenGrok has a concept of Projects - one project is one directory underneath
72SRC_ROOT directory which usually contains a checkout of a project sources.
73(this can be branch, version, ...) 
74
75Projects effectively replace the need to have more web applications, each with
76opengrok .war file. Instead it leaves you with one indexer and one web
77application serving multiple source code repositories - projects.
78Then you have a simple update script and simple index refresher script in
79place, which simplifies management of more repositories.
80
81A nice concept is to have a naming convention for directories underneath
82SRC_ROOT, thereby creating a good overview of projects (e.g.
83name-version-branch).
84
85For example, the SRC_ROOT directory can contain the following directories:
86
87  openssl-head
88  openssl-0.9.8-stable
89  openssl-1.0.0-stable
90
91Each of these directories was created with 'cvs checkout' command (with
92appropriate arguments to get given branch) and will be treated by OpenGrok
93as a project.
94
95
964. OpenGrok install
97-----------------
98
994.1 Installing on Solaris from *.p5p file
100-----------------
101
1024.1.0 Install
103-----------------
104
105The file <package_name>.p5p you can easily use as a new publisher for the pkg command.
106
107  # pkg install --no-refresh -g /path/to/file/<package_name>.p5p opengrok
108
109
1104.1.1 Update
111-----------------
112
113You can also update OpenGrok software with the *.p5p file by running a command
114
115  # pkg update --no-refresh -g /path/to/file/<package_name>.p5p 'pkg://<publisher>/*'
116
117
1185. OpenGrok setup
119-----------------
120
121To setup OpenGrok it is needed to prepare the source code, let OpenGrok index
122it and start the web application.
123
1245.1 Setting up the sources
125--------------------------
126
127Source base should be available locally for OpenGrok to work efficiently.
128No changes are required to your source tree. If the code is under source
129control management (SCM) OpenGrok requires the checked out source tree under
130SRC_ROOT.
131
132By itself OpenGrok does not perform the setup of the source code repositories
133or sychronization of the source code with its origin. This needs to be done by
134the user or by using automatic scripts.
135
136It is possible for SCM systems which are not distributed (Subversion, CVS)
137to use a remote repository but this is not recommended due to the performance
138penalty. Special option when running the OpenGrok indexer is needed to enable
139remote repository support ("-r on").
140
141In order for history indexing to work for any SCM system it is necessary
142to have environment for given SCM systems installed and in a path accessible
143by OpenGrok.
144
145Note that OpenGrok ignores symbolic links.
146
147If you want to skip indexing the history of a particular directory
148(and all of it's subdirectories), you can touch '.opengrok_skip_history' file
149at the root of that directory.
150
151
1525.2 Using Opengrok shell wrapper script to create indexes
153---------------------------------------------------------
154
155For *nix systems there is a shell script called OpenGrok which simplifies most
156of the tasks. It has been tested on Solaris and Linux distributions.
157
1585.2.1 - Deploy the web application
159----------------------------------
160
161First please change to opengrok directory where the OpenGrok shell script is
162stored (can vary on your system).
163
164Note that now you might need to change to user which owns the target 
165directories for data, e.g. on Solaris you'd do:
166
167  # pfexec su - webservd
168  $ cd /usr/opengrok/bin
169
170and run 
171
172  $ ./OpenGrok deploy
173
174This command will do some sanity checks and will deploy the source.war in
175its directory to one of detected web application containers.
176Please follow the error message it provides.
177
178If it fails to discover your container, please refer to optional steps on
179changing web application properties below, which explains how to do this.
180
181Note that OpenGrok script expects the directory /var/opengrok to be
182available to user running opengrok with all permissions. In root user case
183it will create all the directories needed, otherwise you have to manually
184create the directory and grant all permissions to the user used.
185
1865.2.2 - Populate DATA_ROOT Directory
187------------------------------------
188
189During this process the indexer will generate OpenGrok XML configuration file
190configuration.xml and sends the updated configuration to your web app.
191
192The indexing can take a lot of time. After this is done, indexer automatically
193attempts to upload newly generated configuration to the web application.
194Most probably you will not be able to use Opengrok before this is done for the
195first time.
196
197Please change to opengrok directory (can vary on your system)
198
199  $ cd /usr/opengrok/bin
200
201and run, if your SRC_ROOT is prepared under /var/opengrok/src
202
203  $ ./OpenGrok index
204
205otherwise (if SRC_ROOT is in different directory) run:
206
207  $ ./OpenGrok index <absolute_path_to_your_SRC_ROOT>
208
209The above command attempts to upload the latest index status reflected into
210configuration.xml to a running source web application.
211Once above command finishes without errors
212(e.g. SEVERE: Failed to send configuration to localhost:2424),
213you should be able to enjoy your opengrok and search your sources using
214latest indexes and setup.
215
216It is assumed that any SCM commands are reachable in one of the components
217of the PATH environment variable (e.g. 'git' command for Git repositories).
218Likewise, this should be maintained in the environment of the user which runs
219the web server instance.
220
221Congratulations, you should now be able to point your browser to
222http://<YOUR_WEBAPP_SERVER>:<WEBAPPSRV_PORT>/source to work with your fresh
223OpenGrok installation! :-)
224
225At this time we'd like to point out some customization to OpenGrok script
226for advanced users.
227A common case would be, that you want the data in some other directory than
228/var/opengrok. This can be easily achieved by using environment variable
229OPENGROK_INSTANCE_BASE.
230
231E.g. if opengrok data directory is /tank/opengrok and source root is
232in /tank/source then to get more verbosity run the indexer as:
233
234  $ OPENGROK_VERBOSE=true OPENGROK_INSTANCE_BASE=/tank/opengrok \
235       ./OpenGrok index /tank/source 
236
237Since above will also change default location of config file, beforehands(or
238restart your web container after creating this symlink) I suggest doing
239below for our case of having opengrok instance in /tank/opengrok :
240
241  $ ln -s /tank/opengrok/etc/configuration.xml \
242      /var/opengrok/etc/configuration.xml 
243
244More customizations can be found inside the script, you just need to
245have a look at it, eventually create a configuration out of it and use
246OPENGROK_CONFIGURATION environment variable to point to it. Obviously such
247setups can be used for nightly cron job updates of index or other automated
248purposes.
249
2505.2.3 Partial reindex
251---------------------
252
253There is inherent time window between after the source code is updated
254(highlighted in step 5.1 above) and before indexer completes. During this
255time window the index does not match the source code. To alleviate this
256limitation, one can kick off update of all source repositories in
257parallel and once all the mirroring completes perform complete reindex.
258This does not really help in case when some of the source code
259repositories are slow to sync, e.g. because the latency to their origin is
260significant, because the overall mirroring process has to wait for all the
261projects to finish syncing before running the indexer. To overcome this
262limitation, the index of each project can be created just after the
263mirroring of this project finishes.
264
265Thus, the overall approach would be:
266
267  1. create initial index of all the source code
268
269     This will produce configuration.xml, optionally by combining the
270     discovered projects with read-only configuration (as specified
271     with READ_XML_CONFIGURATION). This step has to be performed only once
272     - during the initial OpenGrok setup.
273
274  2. mirror and index all projects in parallel
275
276     This is done by running indexpart command of the OpenGrok script and
277     specifying the configuration.xml written in previous step as
278     READ_XML_CONFIGURATION. The configuration will help the indexer to
279     discover source/data root and project to source path mapping.
280
281  3. perform complete reindex (like in step 1)
282
283     Once all the pre-existing projects are mirrored and indexed, run full
284     indexer to discover projects which have been added or deleted.
285     This will produce new configuration.xml.
286
287When running the indexer the logs are being written to single file. Since
288multiple indexers are being run in parallel in step 2, their logs have to
289be separated. To do this, create logging.properties file for each project
290using the /var/opengrok/logging.properties file as template. The only line
291which can differ would be this:
292
293java.util.logging.FileHandler.pattern = /var/opengrok/log/myproj/opengrok%g.%u.log
294
295Note the path component 'myproj' which separates the logs for given
296project to this directory. The creation of the per-project directory and the
297logging.properties file can be easily done in a script.
298
299The command used in step 2 can look like this:
300
301  OPENGROK_LOGGER_CONFIG_PATH=/var/opengrok/myproj.logging \
302     READ_XML_CONFIGURATION=/var/opengrok/etc/configuration.xml \
303     OpenGrok indexpart /myproj
304
305The last argument is path relative to SRC_ROOT.
306
3075.3 Using SMF service (Solaris) to maintain OpenGrok indexes
308------------------------------------------------------------
309
310If you installed OpenGrok from the OSOLopengrok package, it will work out of
311the box. Should you need to configure it (e.g. because of non-default SRC_ROOT
312or DATA_ROOT paths) it is done via the 'opengrok' property group of the
313service like this:
314
315  # svccfg -s opengrok setprop \
316       opengrok/srcdir="/absolute/path/to/your/sourcetree"
317  # svccfg -s opengrok setprop opengrok/maxmemory="2048"
318
319Then make the service start the indexing, at this point it would be nice if 
320the web application is already running.
321
322Now enable the service:
323
324  # svcadm enable -rs opengrok
325
326Note that this will enable tomcat6 service as dependency.
327
328When the service starts indexing for first time, it's already enabled and
329depending on tomcat6, so at this point the web application should be 
330already running.
331
332Note that indexing is not done when the opengrok service is disabled.
333
334To rebuild the index later (e.g. after source code changed) just run:
335
336  # svcadm refresh opengrok
337
338The service makes it possible to supply part of the configuration via the
339'opengrok/readonly_config' service property which is set to
340/etc/opengrok/readonly_configuration.xml by default.
341
342Note: before removing the package please disable the service.
343If you don't do it, it will not be removed automatically.
344In such case please remove it manually.
345
3465.4 Using command line interface to create indexes
347--------------------------------------------------
348
349There are 2 (or 3) steps needed for this task.
350
3515.4.1 - Populate DATA_ROOT Directory
352------------------------------------
353
354Option 1. OpenGrok: There is a sample shell script OpenGrok that is suitable
355for using in a cron job to run regularly. Modify the variables in the script
356to point appropriate directories, or as the code suggests factor your local
357configuration into a separate file and simplify future upgrades.
358
359Option 2. opengrok.jar: You can also directly use the Java application. If
360the sources are all located in a directory SRC_ROOT and the data and
361hypertext files generated by OpenGrok are to be stored in DATA_ROOT, run
362
363     $ java -jar opengrok.jar -s $SRC_ROOT -d $DATA_ROOT
364
365See opengrok.jar manual below for more details.
366
3675.4.2 - Configure and Deploy source.war Webapp
368----------------------------------------------
369
370To configure the webapp source.war, look into the parameters defined in
371web.xml of source.war file and change them (see note1) appropriately.
372
373    * HEADER: is the fragment of HTML that will be used to display title or
374              logo of your project
375    * SRC_ROOT: absolute path name of the root directory of your source tree
376    * DATA_ROOT: absolute path of the directory where OpenGrok data
377                 files are stored
378
379  - Header file 'header_include' can be created under DATA_ROOT.
380    The contents of this file file will be appended to the header of each
381    web page after the OpenGrok logo element.
382  - Footer file 'footer_include' can be created under DATA_ROOT.
383    The contents of this file file will be appended to the footer of each
384    web page after the information about last index update.
385  - The body of the home page can be changed by updating index_body.html
386    under the webapp directory.
387
3885.4.3 - Path Descriptions (optional)
389------------------------------------
390
391OpenGrok can use path descriptions in various places (e.g. while showing
392directory listings or search results). Example descriptions are in paths.tsv
393file (delivered as /usr/opengrok/doc/paths.tsv by OpenGrok package on Solaris).
394The paths.tsv file is read by OpenGrok indexing script from the configuration
395directory (the same where configuration.xml is located) which will create file
396dtags.eftar in the index subdirectory under DATA_ROOT directory which will
397then be used by the webapp to display the descriptions.
398
399The file contains descriptions for directories one per line. Path to the
400directory and its description are separated by tab. The path to the directory
401is absolute path under the SRC_ROOT directory.
402
403For example, if the SRC_ROOT directory contains the following directories:
404
405foo
406bar
407bar/blah
408random
409random/code
410
411then the paths.tsv file contents can look like this:
412
413/foo	source code for foo
414/bar	source code for bar
415/bar/blah	source code for blah
416
417Note that only some paths can have a description.
418
4195.4.4 - Changing webapp parameters (optional)
420---------------------------------------------
421
422web.xml is the deployment descriptor for the web application. It is in a Jar
423file named source.war, you can change it as follows:
424
425    * Option 1: Unzip the file to TOMCAT/webapps/source/ directory and
426     change the source/WEB-INF/web.xml and other static html files like
427     index.html to customize to your project. 
428    
429    * Option 2: Extract the web.xml file from source.war file
430
431     $ unzip source.war WEB-INF/web.xml
432
433     edit web.xml and re-package the jar file. 
434
435     $ zip -u source.war WEB-INF/web.xml
436
437     Then copy the war files to <i>TOMCAT</i>/webapps directory.
438
439    * Option 3: Edit the Context container element for the webapp
440
441     Copy source.war to TOMCAT/webapps
442
443     When invoking OpenGrok to build the index, use -w <webapp> to set the 
444     context. If you change this(or set using OPENGROK_WEBAPP_CONTEXT) later, 
445     FULL clean reindex is needed.
446
447     After the index is built, there's a couple different ways to set the
448     Context for the servlet container:
449     - Add the Context inside a Host element in TOMCAT/conf/server.xml
450
451     <Context path="/<webapp>" docBase="source.war">
452        <Parameter name="DATA_ROOT" value="/path/to/data/root" override="false" />
453        <Parameter name="SRC_ROOT" value="/path/to/src/root" override="false" />
454        <Parameter name="HEADER" value='...' override="false" />
455     </Context>
456
457     - Create a Context file for the webapp
458
459     This file will be named `<webapp>.xml'.
460
461     For Tomcat, the file will be located at:
462     `TOMCAT/conf/<engine_name>/<hostname>', where <engine_name>
463     is the Engine that is processing requests and <hostname> is a Host
464     associated with that Engine.  By default, this path is
465     'TOMCAT/conf/Catalina/localhost' or 'TOMCAT/conf/Standalone/localhost'.
466
467     This file will contain something like the Context described above.
468
4695.4.5 Custom ctags configuration
470--------------------------------
471
472To make ctags recognize additional symbols/definitions/etc. it is possible to
473specify configuration file with extra configuration options for ctags.
474
475This can be done by setting OPENGROK_CTAGS_OPTIONS_FILE environment variable
476when running the OpenGrok shell script (or directly with the -o option for
477opengrok.jar). Default location for the configuration file in the OpenGrok
478shell script is etc/ctags.config under the OpenGrok base directory (by default
479the full path to the file will be /var/opengrok/etc/ctags.config).
480
481Sample configuration file for Solaris code base is delivered in the doc/
482directory.
483
4845.5 Using Java DB for history cache
485-----------------------------------
486
487By default OpenGrok stores history indexes in gzipped xml files. An alternative
488is to use Java DB instead. This has the advantage of less disk space and
489incremental indexing. Also, for some Source Code Management systems the
490History view will show a list of files modified with given change.
491On the other hand it consumes more system memory because the database has to
492run in background.
493
494Versions of Java DB from 10.5.3 to 10.8.3.0 are known to work.
495Java DB 10.10.x.y versions are known to NOT work.
496To install it perform the following steps:
497
498Solaris 11:
499
500   # pkg install library/java/javadb
501
502Debian/Ubuntu:
503
504  # apt-get install libderby-java
505
506Other:
507
508  Fetch the db-derby bundle from db.apache.org, and unpack to your preferred path.
509
5101) Start the server:
511
512  There are two modes, having Java DB embedded, or running a Java DB server.
513  Java DB server is the default option, we will not describe how to set up the
514  embedded one.
515
516  Solaris 11:
517
518    Use one of the following methods to start the database:
519  
520    a) via SMF (preferred):
521  
522       # svcadm enable javadb
523  
524    b) from command line:
525  
526       $ mkdir -p $DATA_ROOT/derby
527       $ java -Dderby.system.home=$DATA_ROOT/derby \
528           -jar /opt/SUNWjavadb/lib/derbynet.jar start
529  
530  Debian:
531
532    $ mkdir -p $DATA_ROOT/derby
533    $ java -Dderby.system.home=$DATA_ROOT/derby \
534          -jar /usr/lib/jvm/java-6-sun/db/lib/derbynet.jar start
535
536
5372) Copy derbyclient.jar to the lib directory 
538
539  The derbyclient.jar is provided with Java DB installation.
540  The lib directory is the one where opengrok.jar is located.
541  E.g. for Tomcat it is located in the WEB-INF directory which was created
542  as a result of deploying the source.war file.
543
544Copy it over from:
545
546  Solaris 11: /opt/SUNWjavadb/lib/derbyclient.jar
547  Debian: /usr/lib/jvm/java-6-sun/db/lib/derbyclient.jar
548
549  For example on Solaris 11 with bundled Java DB and Tomcat and OpenGrok
550  installed from the OSOLopengrok package the command will be:
551
552    # cp /opt/SUNWjavadb/lib/derbyclient.jar \
553          /var/tomcat6/webapps/source/WEB-INF/lib/
554    # cp /opt/SUNWjavadb/lib/derbyclient.jar \
555          /usr/opengrok/lib
556
5573) Use these options with indexer when indexing/generating the configuration:
558   -D -H
559
560   This is achieved by setting the OPENGROK_DERBY environment variable when
561   using the OpenGrok shell script.
562
563The Java DB server has to be running during indexing and for the web
564application.
565
566Note: To use a bigger database buffer, which may improve performance of both
567indexing and fetching of history, create a file named derby.properties in
568the JavaDB data directory and add this line to it:
569
570  - If using specific data directory:
571
572    # echo "derby.storage.pageCacheSize=25000" >> \
573          $DATA_ROOT/derby/derby.properties
574
575  - Using default directory on Solaris with JavaDB being run from SMF:
576
577    # echo "derby.storage.pageCacheSize=25000" >> \
578          /var/lib/javadb/data/derby.properties
579
5805.6 Introduce own mapping for an extension to analyzer
581------------------------------------------------------
582
583OpenGrok script doesn't support this out of box, so you'd need to add it there.
584Usually to StdInvocation() function after line -jar ${OPENGROK_JAR} .
585It would look like this:
586-A cs:org.opensolaris.opengrok.analysis.PlainAnalayzer 
587(this will map extension .cs to PlainAnalyzer) 
588You should even be able to override OpenGroks analyzers using this option.
589
590
5916. Optional Command Line Interface Usage
592----------------------------------------
593
594You need to pass location of project file + the query to Search class, e.g.
595for fulltext search for project with above generated configuration.xml you'd
596do:
597
598  $ java -cp ./opengrok.jar org.opensolaris.opengrok.search.Search -R \
599        /var/opengrok/etc/configuration.xml -f fulltext_search_string
600
601 For quick help run:
602
603  $ java -cp ./opengrok.jar org.opensolaris.opengrok.search.Search
604
6057. Change web application properties or name
606--------------------------------------------
607
608You might need to modify the web application if you don't store the
609configuration file in the default location
610(/var/opengrok/etc/configuration.xml).
611
612To configure the webapp source.war, look into the parameters defined in
613WEB-INF/web.xml of source.war (use jar or zip/unzip or your preferred zip
614tool to get into it - e.g. extract the web.xml file from source.war ($ unzip
615source.war WEB-INF/web.xml) file, edit web.xml and re-package the jar file
616(zip -u source.war WEB-INF/web.xml) ) file and change those web.xml
617parameters appropriately. These sample parameters need modifying(there are
618more options, refer to manual or read param comments).
619
620    * CONFIGURATION - the absolute path to XML file containing project
621    * configuration (e.g. /var/opengrok/etc/configuration.xml )
622    * ConfigAddress - port for remote updates to configuration, optional,
623    * but advised(since there is no authentication) to be set to
624    * localhost:<some_port> (e.g. localhost:2424), if you choose some_port
625    * below 1024 you have to have root privileges
626
627If you need to change name of the web application from source to something
628else you need to use special option -w <new_name> for indexer to create
629proper xrefs, besides changing the .war file name. Be sure that when this
630changed you reindex cleanly from scratch. Examples below show just
631deploying source.war, but you can use it to deploy your new_name.war too.
632
633Deploy the modified .war file in glassfish/Sun Java App Server:
634
635  * Option 1: Use browser and log into glassfish web administration interface
636
637    Common Tasks / Applications / Web Applications , button Deploy and point
638    it to your source.war webarchive
639
640  * Option 2: Copy the source.war file to
641    GLASSFISH/domains/YOURDOMAIN/autodeploy directory, glassfish will try
642    to deploy it "auto magically".
643    
644  * Option 3: Use cli from GLASSFISH directory:
645
646    # ./bin/asadmin deploy /path/to/source.war
647
648Deploy the modified .war file in tomcat:
649
650  * just copy the source.war file to TOMCAT_INSTALL/webapps directory.
651
6528. OpenGrok systray
653-------------------
654
655The indexer can be setup with agent and systray GUI control application.
656This is optional step for those who wish to monitor and configure OpenGrok
657from their desktop using systray application.
658
659An example opengrok-agent.properties file is provided, which can be used when
660starting special OpenGrok Agent, where you can connect with a systray GUI
661application.
662
663To start the indexer with configuration run:
664
665  $ java -cp ./opengrok.jar org.opensolaris.opengrok.management.OGAgent \
666        --config opengrok-agent.properties
667
668Then from the remote machine one can run:
669
670  $ java -cp ./opengrok.jar \
671        org.opensolaris.opengrok.management.client.OpenGrokTrayApp
672
673assuming configuration permits remote connections (i.e. not listening on
674localhost, but rather on a physical network interface).
675
676This agent is work in progress, so it might not fully work.
677
6789. Information for developers
679-----------------------------
680
6819.0 Building
682------------
683
684Just run 'ant' from command line in the top-level directory or use build
685process driven by graphical developer environment such as Netbeans.
686
687Note: in case you are behind http proxy, use ANT_OPTS to download jflex, lucene
688E.g. $ ANT_OPTS="-Dhttp.proxyHost=?.? -Dhttp.proxyPort=80" ant
689
690
6919.0.1 Package build
692-------------------
693
694Run 'ant package' to create package (specific for the operating system this is
695being executed on) under the dist/ directory.
696
6979.1 Unit testing
698----------------
699
700Note: For full coverage report your system has to provide proper junit test 
701environment, that would mean:
702
703  - you have to use Ant 1.9 and above
704  - at least junit-4.12.jar has to be in ant's classpath (e.g. in ./lib)    
705
706  - install derby.jar to ant's classpath so that Java DB tests can be run
707  - your PATH must contain directory with exuberant ctags binary
708    - Note: make sure that the directory which contains exuberant ctags binary
709      is prepended before the directory with plain ctags program.
710  - your PATH variable must contain directories which contain binaries of
711    appropriate SCM software which means commands hg, sccs, cvs, git, bzr, svn
712    (svnadmin too). They must be available for the full report.
713
714The tests are then run as follows:
715
716  $ ant -lib ./lib test
717
718To check if the test completed without error look for AssertionFailedError
719occurences in the TESTS-TestSuites.xml file produced by the test run.
720
7219.2 Using Findbugs
722------------------
723
724If you want to run Findbugs (http://findbugs.sourceforge.net/) on OpenGrok,
725you have to download Findbugs to your machine, and install it where you have 
726checked out your OpenGrok source code, under the lib/findbugs directory,
727like this:
728
729  $ cd ~/.ant/lib
730  $ wget http://..../findbugs-x.y.z.tar.gz
731  $ gtar -xf findbugs-x.y.z.tar.gz
732  $ mv findbugs-x.y.z findbugs
733
734You can now run ant with the findbugs target:
735
736  $ ant findbugs
737  ...
738  findbugs:
739   [findbugs] Executing findbugs from ant task
740   [findbugs] Running FindBugs...
741   [findbugs] Warnings generated: nnn
742   [findbugs] Output saved to findbugs/findbugs.html
743
744Now, open findbugs/findbugs.html in a web-browser, and start fixing bugs !
745
746If you want to install findbugs some other place than ~/.ant/lib, you can
747untar the .tar.gz file to a directory, and use the findbugs.home property to
748tell ant where to find findbugs, like this (if you have installed fundbugs
749under the lib directory):
750
751  $ ant findbugs -Dfindbugs.home=lib/findbug
752
753There is also a findbugs-xml ant target that can be used to generate XML files
754that can later be parsed, e.g. by Jenkins.
755
7569.3 Using Jacoco
757--------------
758
759If you want to check test coverage on OpenGrok, download jacoco from
760http://www.eclemma.org/jacoco/. Place jacocoagent.jar and jacocoant.jar in the
761opengrok/lib ~/.ant/lib or into classpath (-lib option of ant).
762
763Now you can instrument your classes and test them run:
764
765  $ ant -Djacoco=true -Djacoco.home=/<path_to>/jacoco jacoco-code-coverage
766
767Now you should get output data in jacoco.exec
768
769Look at jacoco/index.html to see how complete your tests are.
770
7719.4 Using Checkstyle
772--------------------
773
774To check that your code follows the standard coding conventions,
775you can use checkstyle from http://checkstyle.sourceforge.net/
776
777First you must download checkstyle from http://checkstyle.sourceforge.net/ ,
778You need Version 6.8 (or newer). Extract the package you have
779downloaded, and create a symbolic link to it from ~/.ant/lib/checkstyle,
780e.g. like this:
781
782   $ cd ~/.ant/lib
783   $ unzip ~/Desktop/checkstyle-6.8.zip
784   $ ln -s checkstyle-6.8 checkstyle
785
786You also have to create symbolic links to the jar files:
787
788   $ cd checkstyle   
789   $ ln -s checkstyle-6.8-all.jar checkstyle-all.jar
790
791To run checkstyle on the source code, just run ant checkstyle:
792
793   $ ant checkstyle
794
795Output from the command will be stored in the checkstyle directory.
796
797If you want to install checkstyle some other place than ~/.ant/lib, you can
798untar the .tar.gz file to a directory, and use the checkstyle.home property
799to tell ant where to find checkstyle, like this (if you have installed 
800checkstyle under the lib directory):
801
802  $ ant checkstyle -Dcheckstyle.home=lib/checkstyle
803
8049.5 Using PMD and CPD
805---------------------
806
807To check the quality of the OpenGrok code you can also use PMD
808from http://pmd.sourceforge.net/.
809
810How to install:
811
812  $ cd ~/.ant/lib
813  $ unzip ~/Desktop/pmd-bin-5.2.3.zip
814  $ ln -s pmd-5.2.3/ pmd
815
816To run PMD on the source code, just run ant pmd:
817
818  $ ant -Dpmd.home=~/.ant/lib/pmd pmd
819
820Output from the command will be stored in the pmd subdirectory:
821
822  $ ls pmd
823  pmd_report.html  pmd_report.xml
824
825If you want to install PMD some other place than ~/.ant/lib, you can
826unzip the .zip file to a directory, and use the pmd.home property
827to tell ant where to find PMD, like this (if you have installed 
828PMD under the ./ext_lib directory):
829
830  $ ant pmd -Dpmd.home=ext_lib/pmd
831
832To run CPD, just use the same as above, but use targets:
833
834  $ ant -Dpmd.home=ext_lib/pmd cpd cpd-xml
835
836Which will result in:
837
838  $ ls pmd
839  cpd_report.xml cpd_report.txt
840
8419.6 Using JDepend
842-----------------
843
844To see dependencies in the source code, you can use JDepend from
845http://clarkware.com/software/JDepend.html.
846
847How to install:
848
849  $ cd ~/.ant/lib
850  $ unzip ~/Desktop/jdepend-2.9.zip
851  $ ln -s jdepend-2.9/ jdepend
852  $ cd jdepend/lib
853  $ ln -s jdepend-2.9.jar jdepend.jar
854
855How to analyze:
856
857  $ ant jdepend
858
859Output is stored in the jdepend directory:
860
861  $ ls jdepend/
862  report.txt  report.xml
863
8649.7 Using SonarQube
865-------------------
866
867Use a sonar runner with included sonar-project.properties properties, 
868e.g. using bash:
869
870  $ cd <checkout_dir> # it has to contain sonar-project.properties!
871  $ export SONAR_RUNNER_OPTS="-Xmx768m -XX:MaxPermSize=256m"
872  $ export SERVERIP=10.163.26.78
873  $ ~//Projects/sonar-runner-2.3/bin/sonar-runner \
874    -Dsonar.host.url=http://${SERVERIP}:9000 
875    -Dsonar.jdbc.url=jdbc:h2:tcp://${SERVERIP}:9092/sonar
876
8779.8 Using Travis CI
878-------------------
879
880Travis depends on updated and working maven build.
881Please see .travis.yml, if your branch has this file,
882you should be able to connect your Github to Travis CI.
883OpenGroks Travis is here: https://travis-ci.org/OpenGrok/OpenGrok
884
885
88610. Tuning OpenGrok for large code bases
887---------------------------------------
888
889While indexing big source repos you might consider using ZFS filesystem to give 
890you advantage of datasets which can be flipped over or cloned when needed.
891If the machine is strong enough it will also give you an option to 
892incrementally index in parallel to having the current sources and index in sync.
893(So tomcat sees certain zfs datasets, then you just stop it, flip datasets to 
894the ones that were updated by SCM/index and start tomcat again - outage is 
895minimal, sources+indexes are ALWAYS in sync, users see the truth)
896
897OpenGrok script by default uses 2G of heap and 16MB per thread for flush size of 
898lucene docs indexing(when to flush to disk).
899It also uses default 32bit JRE.
900This MIGHT NOT be enough. You might need to consider this:
901Lucene 4.x sets indexer defaults:
902 DEFAULT_RAM_PER_THREAD_HARD_LIMIT_MB = 1945;
903 DEFAULT_MAX_THREAD_STATES = 8;
904 DEFAULT_RAM_BUFFER_SIZE_MB = 16.0; 
905 - which might grow as big as 16GB (though DEFAULT_RAM_BUFFER_SIZE_MB shouldn't
906 really allow it, but keep it around 1-2GB)
907
908 - the lucenes RAM_BUFFER_SIZE_MB can be tuned now using the parameter -m, so 
909running a 8GB 64 bit server JDK indexer with tuned docs flushing(on Solaris 11):
910
911 # export JAVA=/usr/java/bin/`isainfo -k`/java
912 (or use /usr/java/bin/amd64/java )
913 # export JAVA_OPTS="-Xmx8192m -server"
914 # OPENGROK_FLUSH_RAM_BUFFER_SIZE="-m 256" ./OpenGrok index /source
915
916Tomcat by default also supports only small deployments. For bigger ones you
917MIGHT need to increase its heap which might necessitate the switch to 64-bit 
918Java. It will most probably be the same for other containers as well.
919For tomcat you can easily get this done by creating conf/setenv.sh:
920
921 # cat conf/setenv.sh
922 # 64-bit Java
923 JAVA_OPTS="$JAVA_OPTS -d64 -server"
924
925 # OpenGrok memory boost to cover all-project searches
926 # (7 MB * 247 projects + 300 MB for cache should be enough)
927 # 64-bit Java allows for more so let's use 8GB to be on the safe side.
928 # We might need to allow more for concurrent all-project searches.
929 JAVA_OPTS="$JAVA_OPTS -Xmx8g"
930
931 export JAVA_OPTS
932
933
934For tomcat you might also hit a limit for http header size (we use it to send 
935the project list when requesting search results):
936 - increase(add) in conf/server.xml maxHttpHeaderSize
937  connectionTimeout="20000"
938 maxHttpHeaderSize="65536"
939  redirectPort="8443" />
940
941Refer to docs of other containers for more info on how to achieve the same.
942
943The same tuning to Apache can be done with the LimitRequestLine directive:
944
945  LimitRequestLine 65536
946  LimitRequestFieldSize 65536
947
948
94911. Authors
950-----------
951
952The project has been originally conceived in Sun Microsystems by Chandan B.N.
953
954Chandan B.N, (originally Sun Microsystems) http://blogs.oracle.com/chandan/
955Trond Norbye, norbye.org
956Knut Pape, eBriefkasten.de
957Martin Englund, (originally Sun Microsystems)
958Knut Anders Hatlen, Oracle. http://blogs.oracle.com/kah/
959Lubos Kosco, Oracle. http://blogs.oracle.com/taz/
960Vladimir Kotal, Oracle. http://blogs.oracle.com/vlad/
961
96212. Contact us
963--------------
964
965Feel free to participate in discussion on discuss@opengrok.java.net.
966
967You can subscribe via web interface on:
968
969  http://java.net/projects/opengrok/lists
970
971