Command "git [command] --help" opens as HTML in my code editor - no output to terminal. How do I revert to default behavior?

This is not the default behavior of git so you have made some local config change to cause this to happen.  Look in your ~/.gitconfig to see if there's anything that doesn't make sense to you.  Try commenting it out.  The core.pager setting is one of the first things I would check.  This stackoverflow question has a variety of options for dealing with this sort of thing.
update 1
I haven't tried to get this sort of thing to work, but according to the docs you can also specify a man.viewer to define the program that is going to run.  Do you get any errors if you try git help -w reset?
One work around would be to get the help to show up in the terminal by using git help -m reset.  By using the help subcommand instead of subcommand --help you can specify whether you want it to try man, info, or web.


What is the difference between Container Engine Software and Container Management Software

I see a lot of confusion in the question. Docker is a containerization software whereas Kubernetes and Mesos are container management softwares.
While Kubernetes is a software for cluster management of containers(not necessarily for docker) and microservices, Apache Mesos is an orchestration technology for cluster management softwares like Kubernetes.

All these three types of technology are aligned to work together.

In layman terms, you (can) run Docker within Kubernetes which you (can) run within Mesos.

Please have a read here for more explaination.


Process guidance for new development team leader

I suspect I'm late to the party, but I've survived a few similar scenarios and might have some tips for you.  It sounds like your team has been over-managed and has grown hostile to process or rules in general.

Do not go directly to Agile and sprint based development.  Consider going with Kanban, and link work items to an epic/larger context.  Sometimes that context will be "break/fix" and sometimes it will be features.  Linking will help you make it clearer where time is spent.  Kanban is preferable because it's likely the absolute minimum process necessary to get work done - it won't require you to bludgeon your team with buzzwords, burndowns, estimates, and other distractions.  It WILL allow you to limit the amount of work in progress - I suggest you start with 3-4 things per team member and try to whittle it down until you can reach a state of continous flow.  Lead your standup using this kanban board.
Look for opportunities to incentivize smaller delivery batches, and run them through an automated testing process.  Use this process to gradually improve the code base and build trust with the release managers who currently introduce friction to your delivery process, which causes an increase in change/batch size, which causes increased risks, which "validates" the need for an outside gate to production.
Break up monolithic repositories, then break up monolithic applications.  Smaller applications again mean smaller batch sizes and reduced risk.
LISTEN a lot.  Your team has been traumatized by a toxic manager, and you have damaged goods on your hands.  If you come in too hot, you'll break trust.  ESPECIALLY through the early phases of this transition, building trust and rapport are the keys to you ever getting a devops practice off the ground.
DO NOT switch tools simply because other people, who seem to be better at this than your team, are using them.  Do I think there are better workflows than your team has?  You bet!  But don't switch for superstitious or personal reasons.  Understand WHAT your current toolchain prevents you from achieving and make sure that the underlying concepts are understood by every single member of your team before you begin.
Be prepared for this to take a year or two to really turn around.  You will need to apply slow, predictable, steady pressure on the organization in order to drive meaningful or longterm change.


Jenkins Select unique parameter with multiple values

My suggestion is to set job parameterized with two options, and just set up variables in the Shell Build step, like this:

if [ "$VAR" == "1" ]; then
    value1="1.0.0"
    value2="myname"
    value3="build/libs"
elif [ "$VAR" == "2" ]; then
    value1="2.0.0"
    value2="myname2"
    value3="var/lib"
fi

echo ${var.value1}
echo ${var.value2}
echo ${var.value3}


Where do I use Ansible and where is Terraform used? ( Strengths & Weaknesses)

I think you have to consider the history of the two tools to answer this question properly:

Ansible was created to manage the installation and maintenance of virtual machines, thus it originally only had the capability to install software into an operating system, perform systems administration tasks within that machine.  Subsequently, it has been extended to be able to create cloud resources in AWS, Azure, GCP and others.
Terraform was created to manage cloud resources, it's effectively a facade layer over the various APIs in AWS, Azure, GCP, etc.  It has subsequently been extended to be able to cover many of the functions that Ansible also covers such as installing software.

My personal workflow is to use both applications for what they are really good at, thus I create infrastructure (Virtual Machines, Clusters, Storage, Networks, etc) in Terraform and then use Ansible to install and configure software as required.

I can, and have in some situations, just used Terraform or Ansible, it's just a little harder as you have to learn how to use one tool in a different way.

Bruce Becker pointed out in a comment that one of the main differences is the fact that Terraform is "eventually consistent", in that it does a good job of understanding the current state and then making changes that converge on a "desired state". This workflow is better for the majority of tasks than Ansible. In Ansible you can do a test run with --check, but this doesn't always take into account cascading changes.


Azure DevOps: How to pass a parameter from a build to a release?

In case anyone ever finds this question: I didn't find a satisfying answer. My workaround was to write my variables to a text file in the build and then publish it, download it in the deployment and publish the values as output variables to a bash task. Looks something like this:
source file_containing_your_vars.txt
echo "##vso[task.setvariable variable=var1;isoutput=true]$var1"

the output variables parameters of this bash would need some reference name, for example job_out and you would then be able to reference the variables from other jobs in that stage like this:
variables['job_out.var1']

If you are running a yaml deployment pipeline (instead of one made in the portal) you can even use job output variable between stages.
For portal pipelines that want to share variables between stages this exists and seems to have helped some people but I didn't get it to work:
https://stackoverflow.com/questions/53042605/how-to-modify-azure-devops-release-definition-variable-from-a-release-task/53060141#53060141


Pometheus-operator - modify Grafana mixin dashboard

You should use https://github.com/helm/charts/tree/master/stable/prometheus-operator helm chart , also you can write Grafana dashboard script that would create dashboard dynamically , can be update as well later on.


Why do so many circleCI configs use `version: 2` for their workflows?

The Workflows version field was used to issue warnings for deprecation or breaking changes during v2 Beta, and it is deprecated as of CircleCI v2.1.
2 is the only valid number besides 1


You aren't actually using file.managed, you are using file.append - so salt is expecting to add onto the end of an existing file. This may be the cause of your issue. Instead try:

/home/vagrant/.bash_profile:
  file.managed:
    - name: /home/vagrant/.bash_profile
    - source: salt://config/user/.bash_profile
    - mode: 0775
    - user: vagrant
    - group: vagrant
    - template: jinja
    - replace: true

Also, be sure you are using the latest version of Salt Stack, 2016.11


ansible question to execute and display multiple commands

One could use -vvv to get the output of the commands while running the playbook:

changed: [localhost] => (item=echo helloworld) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": [
        "echo",
        "helloworld"
    ],
    "delta": "0:00:00.002517",
    "end": "2019-12-20 15:20:09.042020",
    "invocation": {
        "module_args": {
            "_raw_params": "echo helloworld",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "item": "echo helloworld",
    "rc": 0,
    "start": "2019-12-20 15:20:09.039503",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "helloworld",
    "stdout_lines": [
        "helloworld"
    ]
}


List workflow runs would be the equivalent API endpoint in Github Actions:
GET /repos/:owner/:repo/actions/workflows/:workflow_id/runs

You can use the status query parameter e.g. to filter for successful runs:
GET /repos/:owner/:repo/actions/workflows/:workflow_id/runs?status=success


Relation of devops with release management

Your use of the term "devops" suggests that you see it as a role.  I humbly suggest that developers that are part of a Scrum team aren't called "Agiles," so DevOps practitioners aren't "devops." :)

Release management is absolutely part of DevOps.  It's one of several specialty engineering roles.  Staffing for the skill of release management varies widely from one company to the next.  The size of the company, the industry, and the size of the IT department all influence how (and if) that skill is isolated to a person, a team, or distributed across IT.

Organizations of medium to large size tend to have dedicated staff that understand the practice and tooling of Continuous Integration.  This includes some systems admin skills, and increasingly includes development skills, at least scripting.  The growth of Infrastructure as Code as a practice is blurring the lines between release managers and developers.


Secure deployments of on-prem artifacts to cloud

Both Azure and AWS will have any container images built off their network pushed to their repositories using https which will encrypt the transfer (e.g. AWS ECR encryption).  Both of these providers offer ways to encrypt the transfer of data into their non-container storage solutions (often via https) and then offer ways to encrypt the data at rest.  Once it is on the cloud provider deploying to k8s can all be done securely within their servers.


monitoring/alerting system which can support very arbitrary data source and logic

Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way to avoid this custom monitoring. While monitoring, alerting and reporting management and configuration can be tedious, there is no easy shortcut for doing it properly and it's probably going to take more time, dedication and resources to do it properly than you think.

Out of the 4 monitoring systems I have used extensively, I must recommend nagios - it seems to be the best suited for configuration management systems and most compatable with DevOps philosophy.

It may sometimes be useful to use a system such as elastalert or similar systems as middleware for alerting/monitoring/reporting, but you will have to judge each situation/monitor on a case-by-case basis, but you will find these tools (and most monitoring systems) have some SDK, API or tool capable of helping you with your monitoring customization. Just choose the easiest to configure system with the lowest ROI and support cost and be sure to stick with it and follow your monitoring migration project to completion. Be sure to consider dedicated resources/personnel for this task if possible.


Company is worried about exposing our PII to Tableau, Can we run Tableau without exposing our data to any 3rd parties?

There are several alternatives:

since you are looking to use Tableau as our main visualization tool, you might try other on-premise alternatives, such as:

Pentaho Community Edition
QlikView on-premise

Also, there is on-premise license for Tableau


I'm pretty sure this Jenkins system property will do the trick:

hudson.util.RingBufferLogHandler.defaultSize

tuning
Since: 1.563
Default:  256
Description: Number of log entries in loggers available on the UI at /log/

You can also custom Configure Loggers for Jenkins or create a logger in Jenkins for troubleshooting and diagnostic information that capture specifics you are looking for.
Additional references:
JENKINS - Logging
JENKINS - Logger Configuration
Or, if you are the admin, you should be able simply to scan the logs on the filesystem. Review what log rotation options and retention you may have.


Does TFS Build have something like a Jenkinsfile?

The strict answer is no, in that there is no direct equivalent to a Jenkinsfile. However, there are Build Process Templates and you can use the Psake Build Automation tool in combination with templates to reduce the barriers to entry for creating a new build, test and deployment build definitions.

It is nowhere near as fluid as Jenkins' Pipelines as Code, AppVeyor's appveyor.yml, CircleCI's circleci.yml and Travis' .travis.yml.  There is a wealth of information on the ALM Rangers Library of tooling and guidance solutions.


In my experience "build step" is synonymous with "compilation", e.g. it's the step that builds a set of output files (binary, etc.).


Migrating from TFVS to GIT using GIT-TFS

As of Azure DevOps Server 2019, migration from Team Foundation Version Control (TFVC) to Git is a simple import command.  However, there are caveats to the migration due to how very different Git is in how it stores changes.

Microsoft has a great write-up for this migration and if you have the newer version of Azure DevOps Server or if you're in the cloud, you don't need git-tfs.

But, a few highlights:

Git is distributed version control, which means that you have a full copy of the files and all historical changes simply by cloning the repository to your machine.  You technically don't need a server, but if you have a team and collaborate, a server is needed to share changes with the team via appropriate Git workflows and branching.
Git is really good at storing text files and will only ever store a single copy of a file with each commit storing a snapshot of the differences in that file over time.  Binary files require storage of the full file for each commit that changes the file.
When binary files need to be stored in a Git repo, it's best to use Git LFS (Large File System).  This allows you to mark files as large/binary and Git will only track a reference to the file, not the actual file itself.  This keeps the repository small and nimble.
When you have an enormous repository (files/history) that is in the 2-4 GB range, you may need the Microsoft-contributed Virtual File System for Git, which Microsoft uses for its enormous 500 GB Windows code-base that developers all around the globe work from.  Contributions from Microsoft and the rest of the open-source community have done a great job of keeping Git commands performance-tuned for larger repos, but VFS for Git is a big leap for repos that cannot be small.

If you are still running an older version of TFS that doesn't yet have the easy import option, that's OK.  Usage of git-tfs is a viable solution and Microsoft also has a great write-up for how to approach this manual migration.  In either case, there will potentially be work involved if your code-base if you stored a lot of binary files in TFVC compared to text/code.

What git-tfs is actually doing is creating a historical clone of your TFVC repository into a local Git repository by replaying the history and creating Git commits.  The tool can be used as a way to bridge a TFVC repo and Git repo, but you are better off fully migrating to Git and removing the TFVC repository.

Once you decide to fully migrate, it's best to do the migration in a few steps locally in the new cloned Git repo before pushing it to the TFS server as the origin.  This allows you to clean up any history of binary files that may be needed.  The BFG Repo-Cleaner can be used for this which can re-write history in the Git repo to remove all but the latest change to files like binaries, thus reducing the physical size of the Git repo on disk.


What are best practices to solve `All files/directories will have '-rwxr-xr-x' permissions` when building docker images on Windows?

It is not currently possible to disable that warning, see discussions on Issue 20397:

  That warning was added, because the Windows filesystem does not have
  an option to mark a file as 'executable'. Building a linux image from
  a Windows machine would therefore break the image if a file has to be
  marked executable.

  For that reason, files are marked executable by default when building
  from a windows client; the warning is there so that you are notified
  of that, and (if needed), modify the Dockerfile to change/remove the
  executable bit afterwards.

In a sense this also means it's practically impossible to actually solve the issue.

What maybe can help a bit is moving the error message from stderr to stdout, released in docker 17.04. See Issue 29856.


How to analyze the file activities due to process?

You can use lsof with the -p flag, like this : lsof -p $PID .


How to "inject" AWS Lambda code into a CloudFormation?

You've got a couple of options:

1. Embed the source code to the CloudFormation template

I do just that in my ec2-start-stop demo. You'll see file ec2-start-stop.template.yml with these lines:

    StartStopLambda:
      Type: AWS::Lambda::Function
      Properties:
        [...]
        Runtime: python3.6
        Code:
          ZipFile:
            Fn::Join:
            [...]
            -
              - "%%{ec2-start-stop.lambda.py}%%"

Then I've got the actual Python script referred in the %%{ec2-start-stop.lambda.py}%% and a simple import-files.py script that somewhat intelligently embeds the python file into the yaml file and creates a valid standalone CloudFormation template with the lambda code in it.

Note that you're limited to 4096 bytes including new lines for your lambda code.

2. Use aws cloudformation package / deploy

If you've got a more complex Lambda that needs external libraries or doesn't fit in the 4kB limit you can use aws cloudformation package and aws cloudformation deploy to facilitate the deployment to CI/CD.

In this case your Lambda resource in the template points to a local directory, e.g. lambda/. The package command then zips up the contents of the directory, uploads to the provided S3 Bucket with a unique name and generates an updated CFN template with the link to the uploaded file. The aws cloudformation deploy command then takes the template, creates a change set and applies the changes, thus updating the Lambda.

I've used both methods in CI/CD pipelines and both work well because the source files are kept in separate files until deployment.

Hope that helps :)


Just browsing through the available options, it looks like the nova meta actions may be available as openstack server set:

$ openstack help server set
usage: openstack server set [-h] [--name <new-name>] [--root-password]
                            [--property <key=value>] [--state <state>]
                            <server>

Set server properties

positional arguments:
  <server>              Server (name or ID)

optional arguments:
  -h, --help            show this help message and exit
  --name <new-name>     New server name
  --root-password       Set new root password (interactive only)
  --property <key=value>
                        Property to add/change for this server (repeat option
                        to set multiple properties)
  --state <state>       New server state (valid value: active, error)

E.g., if I run:

openstack server set myserver --property foo=bar

And then run:

openstack server show myserver

I see:

[...]
| properties                           | foo='bar'                                                |
[...]


How to move a containerized application to a different host?

From what I understand of https://stackoverflow.com/questions/28734086/how-to-move-docker-containers-between-different-hosts, docker export will only save the container filesystem. But no metadata such as the default command, leading to that error when you try to run the imported archive:

sh# IMAGE=$(docker import phpbb.tar)
sh# docker run "${IMAGE}"
docker: Error response from daemon: No command specified.

Given my need, it is sufficient to docker commit the container to a new image, then docker save the image, and finally docker load and docker run it on the new host:

old# docker commit CONTAINER name:tag
old# docker save name:tag |gzip > archive.tar.gz
old# scp archive.tar.gz root@new:.

new# docker load < archive.tar.gz
new# docker run name:latest

It is important to save the image by repository name:tag and not by id to preserve the tag in the archive.


How to install a private managed Kubernetes on-premise?

Take a look at OpenShift container platform (https://www.openshift.com/products/container-platform/). They seem to provide on-premise options. You might want to start with OpenShift Origin - the community edition (https://github.com/openshift/origin) to test for free if it fits your plan.


Why does my Terraform config require variables in multiple places?

Each module has its own set of input variables.

The input variables for the root module are set when you run Terraform, e.g. on the command like with the -var option.

The input variables for any child module, like your module "logs" here, are set inside the module block.

Each variable declaration is valid only for the module it appears in. You happen to have given both of these variables the same name, which is fine and sensible as it helps a human reader understand that they represent the same concept, but Terraform considers these two variables entirely separate and requires you to declare and set both of them.

As an analogy, consider a module as being somewhat equivalent to a function in a programming language. Each function defines its own arguments, and two functions might have an argument of the same name but they are still distinct by virtue of belonging to different functions:

function main(env: string) {
  logs(env)
}

function logs(env: string) {
  // "env" in here is "test" because it was passed through
  // the arguments of both of these functions, not because
  // the arguments happened to have the same name in both
  // functions.
}

// Calling main here is, for the purposes of this analogy,
// corresponding to running: terraform apply -var="env=test"
main("test")


Dump Kubernetes log files for all services/pods/jobs etc

You are possibly looking for a third party stack on cluster to help you, if so there is long list you can explore and play to your need
Various Kubernetes logging tools you could consider from below list:

Fluentd
ELK Stack
Stackdriver

I have used this link to setup EFK on my local cluster.


How to create custom AWS CloudWatch metric for i-node

AWS have published a plugin for collectd which allows to send metrics collected using collectd to CloudWatch.

https://github.com/awslabs/collectd-cloudwatch

The collectd monitoring tool runs on the server and can be configured to collect various metrics, among them metrics on disks and filesystems using their DF plugin. https://collectd.org/wiki/index.php/Plugin:DF

You can turn the ReportInodes option of the CollectD DF plugin and then have these metrics reported to CloudWatch as custom metrics using the AWS CollectD CloudWatch plugin.


Default StorageClass set up for Docker EE Kubernetes

Found out that there is no specific 'default' storageclass(SC) type for Docker EE or others, it depends on the requirement and specific to Kubernetes provider(s). I went ahead and created a sc with 'local' provisioner type.


What is the difference between Jenkin's `Replay` and `Rebuild` options?

Rebuild, using the Rebuilder plugin allows a completed job to be re-executed while allowing you to change the parameters, assuming your job is parameterized.
Replay does something similar, except it doesn't show you parameters. Instead, it shows the pipeline code directly. This is helpful when wanting to test changes iteratively as you can click Replay, change one or two lines of code, then let it execute. The job will run using the same parameters as it did the first time it ran/completed.


Instead of trying to find a solution in git, I would recommending using the Lockable Resources Plugin, which is an official plugin developed by the Jenkins developers.  This plugin implements locks in Jenkins, which are basically the same idea as a semaphore or mutex.  Using this plugin, each of your jobs that needs to push to the same branch of the same repo will request a lock when performing the push.  If the lock is available, the job will acquire the lock, push right away, and then release the lock when it is finished pushing.  If the lock is not available, the job will wait until the lock is available (i.e., all other jobs are done pushing) before grabbing it, pushing, and releasing the lock.


Change variable inside of project. Jenkins, docker, kubernetes

As @Harith mentioned, the use of secrets simplifies deployments in various environments using environmental variables. To load in a secret directly use:

  $ kubectl create secret generic < NAME OF SECRET > --from-literal=< KEY >=abc123!:?

NOTE! This will put the secret in the Default namespace, add a namespace flag for specific resources e.g.:

  $ kubectl create secret generic mongodbsecret --from-literal=mongoUser=abc123!:? -n mongoNamespace

You can then reference this secret in the deployment.yml or pod.yml or wherever you put your image for deployment on the kubernetes cluster.

env:
- name: MONGO_INITDB_ROOT_USERNAME <name of the env var in the code>
   valueFrom:
     secretKeyRef:
       name: mongodbSecret <NAME OF SECRET>
       key: mongoUser <KEY>

However, if all you need is to change 'development' to 'production' you could put an environment var directly in the (deployment).yaml.

env:
- name: NODE_ENV
  value: production

You can also update the Jenkinsfile to send in the environment by for instance:

def resource_env = BRANCH_NAME == 'master' ? 'production' : 'development'

sh "kubectl set env ${kubeImageName} NODE_ENV=${resource_env} --namespace ${kubeNamespace}"

On your current pipeline idea: The safest place for sensitive env vars would be putting it in kubernetes in a secret in your situation. However, for simple selection of deployment environments then using a Jenkinsfile simplifies things a bit like 'prod' vs 'dev'. Putting sensitive, variable or private things on git is a bad idea. Keep your docker images precise and task-orientated.

Hope that helps


How do I enable Google App Engine API with Terraform?

The terraform service account was located in a separate project and the API needed to be enabled for appengine in the account the Terraform service account runs in even though it was creating a resource in an another project.


salt-stack: capture group for regex include_pat

A capture group is a type of backreference. Saltstack, being python based, uses the re.match object to do regular expression matching, which appears to support backreferences, so these backreferences are allowed.
However, you be unable to pass the value of the backreference \1 in the - include_pat field back to - name because the - name field is never processed by the re.match object. Even if it were, variable scope would prevent you from sharing the backreference between re.match calls.
Instead, you will need to do one of the following:

Rename all the files in - source from myfile.conf.template to myfile.conf

Use cmd.run to do a batch file rename after your file.recurse and add to your file.recurse an onchanges pointed to a previous event in order to defeat the idempotency (that is, to prevent it from writing out the .template files every time salt runs). You will also be sure to use a require and onchanges in your cmd.run that references copy_template
This will have the unfortunate side-effect that you will never be able to update your jinja templates. You will only be able to deploy them the first time - at least not without first deleting what is there, and then re-writing and re-copying from the source.

Use file.managed for every template file.

For example:
copy_template1:
  file.managed:
    - name: '{{ service_path }}\template1.conf'
    - template: jinja
    - source: {{ pillar['locations']['systemtoolsfolder'] }}\template1.conf.template
copy_template2:
  file.managed:
    - name: '{{ service_path }}\template2.conf'
    - template: jinja
    - source: {{ pillar['locations']['systemtoolsfolder'] }}\template2.conf.template
...
and so forth


Create User Using Ansible with Authenticate Pem File

I suspect what is happening here is you are trying to insert the private key into the authorized_keys file, which is invalid as only the public key is required on the target machine.

What you need to do is extract the public key from the private key:

- name: Generate an OpenSSL public key with a passphrase protected private key
  openssl_publickey:
    path: /tmp/key-{{ item.username }}.pub
    privatekey_path: {{ item.pem_file_path }}
    privatekey_passphrase: ansible
    format: OpenSSH
  with_items:
    - "{{ users }}"

and then use the public key:

- name: Set Authorized key token from the file
  become: true
  authorized_key:
    user: "{{ item.username }}"
    state: present
    key: "{{ lookup('file', '/tmp/key-{{ item.username }}.pub') }}"
  with_items:
    - "{{ users }}"


Conditionally stopping SQL replication during deployment

I eventually found the solution :)

In an Azure Devops Release pipelines I've created a Command Line task that executes SqlPackage.exe /Action:DeployReport /p:DoNotAlterReplicatedObjects=false ... which creates an XML file that contains info on which tables etc would be changed by a deployment. It's important to note that DoNotAlterReplicatedObjects should be set to false, otherwise SqlPackage will fail.

In a second task in the Azure Release pipeline, I analyze the generated DeployReport (e.g. the XML file) using Powershell and XPath**. The Powershell task will create an boolean output variable that will indicate whether replication needs to be stopped or not. In my case, I analyze the XML and if a table is created/altered/dropped, the replication will be stopped.
The syntax for setting the output variable is

Write-Host "##vso[task.setvariable variable=haltReplication;isOutput=true]$haltReplication"`

Third step is yet another Powershell script that will stop the replication. This third script will only run if $haltReplication evaulates to true. In the Control Options, the custom condition is and(succeeded(), eq(variables['haltReplication.haltReplication'],'1'))

**The XML file generated by SqlPackage.exe contains namespaces, which makes working with XPath a bit cumbersome. The following SO answer helped me a lot https://stackoverflow.com/questions/36417189/ignore-namespace-in-xpath-as-this-can-be-dynamic?rq=1


What are best practices for sharing and accessing Vault secrets?

Assuming person B has a role which also allows them access to the path secret/hello, then they would simply run vault read secret/hello from their own machine.

If you're following the demo, I think the vault server will be on your local machine? So if someone else is to connect to it they would need to set VAULT_ADDR to point to the IP where the server is running.


How to specify specific subnet in Terraform when using for each

When you want to refer to a specific instance of a resource that has multiple instances due to using for_each, you need to include the specific key of the instance you want in your references:

  subnet_id = aws_subnet.private["us-east-1a"].id

If you just want to select any one subnet from the list, without needing to specify its name directly, you can use an expression like the following to select a single key by sorting the keys lexically and taking the first one:

  subnet_id = aws_subnet.private[keys(aws_subnet.private)[0]].id

However, beware that if you do the above then any later changes to var.subnet_numbers_private that affect which subnet sorts first will cause a different subnet id to be selected, which will then in turn cause Terraform to plan to replace your instance to move it into a new subnet.

From the way you wrote the question it sounds like you want just one instance in a single subnet rather than one instance per subnet, but if you did want an instance per subnet then you can achieve that by putting for_each in the aws_instance resource too, using the subnet resource itself (represented in expressions as a map) as the repetition expression:

resource "aws_instance" "jenkins" {
  for_each = aws_subnet.private

  # ...
  subnet_id = each.value.id
  # ...
}

In this latter case, the aws_instance.jenkins instances will also be identified by the keys you selected for the subnets, so you'd have instances with addresses like aws_instance.jenkins["us-east-1a"] and aws_instance.jenkins["us-east-1b"] so both you and Terraform can see which instance belongs to which subnet.


While you can benchmark this yourself, benchmark testing has shown that there isn't much of a difference in time between long and short polling when the queue is full.  Additionally, with long polling you always have data coming back unlike short polling.

As for when to use short polling the AWS FAQ covers this:

  For example, if your application uses a single thread to poll multiple queues, switching from short polling to long polling will probably not work, because the single thread will wait for the long-poll timeout on any empty queues, delaying the processing of any queues that might contain messages.

Additionally:

  ..if your application expects an immediate response from a ReceiveMessage call, you might not be able to take advantage of long polling without some modifications to your application.

If you don't want to deal with asynchronous queues and multiple threads handling the queues in your application then short polling may provide an easier development path (although you'll have to deal with empty responses).


What is the Kubernetes version? Is it >1.8 ? (which has RBAC enabled by default).

The error message says that the Kubernetes Dashboard service account is not allowed to list configmaps in default namespace. RBAC by default is deny-all. You need to explicitly grant permissions.

You can create the following ClusterRoleBinding to grant dashboard full admin privileges.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

Save the above block into dashboard-rolebinding.yaml and create it on the cluster:

kubectl create -f dashboard-rolebinding.yaml

IMPORTANT: Make sure your dashboard is secured from outside before doing this. Otherwise anyone can have complete access to your cluster. Here is a great article about securing  kubernetes dashboard, from Heptio: https://blog.heptio.com/on-securing-the-kubernetes-dashboard-16b09b1b7aca writted by Joe Beda himself.


NOTE: It's probably not worth reading too much into the outward comprehension of how good-or-not StackExchange is at managing their disaster recovery scenarios.  I suspect they are following much of the best practice below and simply testing scenarios to validate their configuration.

Depending on the environment you operate within:

A disaster recovery plan might form part of a larger Business Continuity Plan, a Business Continuity Plan would also likely consider operational risks to you people, organisation, locations, information, partners and management systems.
A disaster recovery plan might be broken down into many IT Service Continuity Plans for individual services.  The disaster recovery plan would likely incorporate people and process along with the technical aspects of the service.

Given these definitions, you might consider ways of improving the capability of the organisation as a whole to be resilient against failure:

Service Recovery:

Enable individual services to be Active-Active across two geographically dispersed data centres.  This does assume that applications are capable of replicating state between data centres, for example using BASE Semantics for data.
Create Self-Healing services, this means expecting failure and building with Resilience Engineering in mind.  An example is by using a tool such as Chaos Monkey to simulate a failure.

Disaster Recovery Plan:

Again enable Active-Active mechanisms across data centres, the difference from SRPs is that capacity needs to be carefully considered, i.e. if you have to DCs in an Active-Active pattern and one fails then a single DC must be sufficiently scaled to support 100% of the traffic.
War Games and Rehearsals are really important for disaster recovery plans as this tests the people and the process, in the most mature DevOps environments much of this can be automated as evidenced by Chaos Gorilla.

Business Continuity Plan:

On the basis that this is a DevOps site I won't go into the long process of building Business Continuity plans.  However, the rules of not keeping all of your eggs in one basket apply - have a plan for when your office is flooded:

Allow your staff to work remotely from home one day a week, this tests your BCP strategies.
If possible have to geographically and politically separate locations for your workforce.
Define and test a clear process for communicating business continuity events and practice them through fire drills.


I've not tested it, but as the doc says, you can configure the plugin via two ways : ini entries or env.

Add below your snippet the following and it should work :

[defaults]
callback_whitelist = profile_tasks

[callback_profile_tasks]
sort_order = descending


Installing ingress controller, services and pods on K8s via Ansible

Since your new in this platform, I would suggest you to explore in the bellow order (assuming you want to use nginx as the load balancer ingress controller):
1- This tutorial is the most complete to get started with k8s. I strongly suggest you to follow it through to understand the jargon and the working components before going to the next steps.
2- Take a look at this example where they go through creating an nginx application in a k8s cluster.
3- Look at this tutorial where the author describes the key concepts and working operation of an ingress controller.
4- Finally, look at this to start setting up your ingress controller from the official doc page. Other resources are available to allow you to setup as much as you'd like.
Extra: Here is the official k8s kubectl cheat sheet.


Dealing with ambiguity of the terms publish vs. deploy

The terms are actually quite well defined and some authors are simply making a mess of things by using a term like 'publish' for deployment, in cases where publishing a website for example constitutes for their own process also a deployment, as there is no extra deployment step after publishing. Sometimes they use word publishing both for uploading and publishing together as all of their artifacts are public. That creates unnecessary confusion. Just to make it clear, here are the definitions:

Environment Definitions:

  build environment - Environment setup for build process.
  runtime environment - Environment setup for execution process (test/staging/production).

Artifact Lifecycle Stages:

  building - process of creating an artifact in build environment
  uploading - moving artifact from build environment to repository.
  publishing - making artifact in repository public.
  downloading - moving artifact from repository to runtime environment
  installing - applying artifact to runtime environment
  deploying - using artifact in runtime environment
  delivery - making artifact (effects) available to customer

Note: People often use any of those terms to mean one or more of the previous terms together.


Gitlab runner, docker registry and self-signed certs

Oh well, the answer was simple: why do you need to use volume only for file transfer? Just volume the directory, containing problematic folder with cert in it.


Re-using Python virtual environments on a build server

It's generally a best practice to make your builds idempotent.  Leaving artifacts behind only opens up opportunities for dependency management issues- and resolving those in a resilient way is exactly why you're using a build server.  I suggest you take a look at WHY your Python builds are running so slow- if you discover this is because of an overabundance of packages in each project, this might be optimizable.  If you discover this is because of server performance issues, those are easily fixable with more resources.  If you find you are blocked by network throughput, consider using an artifact repository to cache packages locally.  But building persistent virtualenvs seems like a recipe for problems that could be unpredictable when moving to another server, and extremely frustrating to troubleshoot.


Can I upgrade a docker image/container from buntu 18.04 to ubuntu 20.04?

Docker doesn't recommend running an apt-get upgrade and dist-upgrade on a docker image as many of the "essential" packages from the partent image cannot upgrade inside an unprevileged container. take a look at the Docker best practices


How to re-enable RDP on AWS-EC2 instance

Yes, You can able to re-enable RDP using run the AWSSupport-ExecuteEC2Rescue Automation (AWS Systems Manager)

Please follow below steps carefully:

Open the AWS Systems Manager console at https://console.aws.amazon.com/systems-manager/.
In the navigation pane, choose Automation.
Choose Execute automation.
In the documents list, choose AWSSupport-ExecuteEC2Rescue. The document owner is Amazon.
In the Document details section verify that Document version is set to the highest default version. For example, 13 (default).
In the Execution mode section, choose Simple execution. Leave the Targets and Rate Control option disabled.
In the Input parameters section, specify the following parameters:

For UnreachableInstanceId, specify the ID of the unreachable instance.

For LogDestination, specify an Amazon S3 bucket if you want to collect operating
system-level logs while troubleshooting your instance. Logs are automatically uploaded
to the specified bucket.

For EC2RescueInstanceType, specify an instance type for the EC2Rescue instance. The
default instance type is t2.small.(Please leave as it is if you are confused and go
ahead)

For SubnetId, specify a subnet in an existing VPC in the same availability zone as the
unreachable instance. By default, Systems Manager creates a new VPC, but you can
specify a subnet in an existing VPC if you want. (Please leave as it is if you are
confused and go ahead)

Note

If you don't see the option to specify a bucket or a subnet ID, verify that you are using the latest Default version of the document.

For AssumeRole, if you created roles for this Automation by using the CloudFormation procedure described earlier in this topic, then specify the AssumeRole ARN that you copied from the CloudFormation console.

Choose Execute automation.

It takes sometimes (5-10 minutes) and you are able to access your old instance with new IP.

Here is the AWS documentation for this issue:
https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-remote-desktop-connection-ec2-windows/


CentOS 7 SSH login with Keys Authentication

Yes, but you will first need to log in as the master or root account, and then sudo su otheruser in order to log in as that user. There is no way to log in as any user without first going through a privleged user's account.

An additional caveat is that the user you are trying to log in as needs to have a shell of /bin/bash or similar valid shell in /etc/passwd. If the user has a shell like /bin/fase or /bin/nologin you will be unable to change to this user's account (but neither will anyone else for that matter).


Which university has a good postgraduate DevOps program?

I'm not really familiar with the topics you mentioned. However I went ahead to go checkout kuleuven.be, i.e its Computer Science departement (with sufficient credibility, trust me).

Here are some interesting articles in which DevOps is mentioned:

Automated workflow regression testing for multi-tenant SaaS: Integrated support in self-service configuration dashboard.

Here is part of the abstract of it:

  ... This effectively makes these tenant administrators part of the DevOps team, and in turn creates the need for the SaaS provider to provide them with Quality Assurance tool support. One such tool is a regression testing framework that allows them to make sure that a new version of a workflow can behave similarly as to a successful execution of a previous version. ...

Variability middleware for multi-tenant SaaS applications.

Here is part of the abstract of it:

  ... The focus has been mainly on (1) the pre-deployment phases of the development life cycle and (2) fine-grained (tenant-level), run-time activation of specific variants. However, with upcoming trends such as DevOps and continuous delivery and deployment, operational aspects become increasingly important. ...

Both articles have links with contact info to their authors.


Pipeline flow get stuck at “Notifying stash” after finishing all stages

We removed the OCP plugin configuration from Jenkins


Is there something like Jenkins pipeline scripts for Bamboo?

I don't think Bamboo has that natively but there are plugins such as this one:  (https://marketplace.atlassian.com/plugins/com.cobalt.cdpipeline.cdpipeline/server/overview).

Atlassian also has a different product called Bitbucket pipeline (https://bitbucket.org/product/features/pipelines) that may be more similar to what you want but it's more of a CircleCI type of Pipeline.


How to store/retrieve multiple ansible-vault passwords within/from a single file

I totally misunderstood you. Here is a new attempt to actually answer your question. :)

Create a passwords file:

root@2ca77340d571:/# cat .passvaults
dev my_dev_pass
test my_test_pass
prod my_prod_pas

Create a file to encrypt:

root@2ca77340d571:/# cat extravaulty.yml
chocolate:banana

Encrypt the file:

root@2ca77340d571:/# ansible-vault encrypt --vault-id=dev@.passvaults extravaulty.yml
Encryption successful

View the raw file:

root@2ca77340d571:/# cat extravaulty.yml
$ANSIBLE_VAULT;1.2;AES256;dev
33336336336238326137393764316333623231336238323931306166626434653164326330656566
3561356333343435663538623661363661373461356461300a663361666331323031623530343930
62633230636566643339663637386438336539383162346634393031633165333033396238363163
6134383233346538330a326266626261346638303738313862656337396237343231326231656139
34376334663664343164303462356431306539316232323761386164643735376330

and the decrypted file:
root@2ca77340d571:/# ansible-vault view --vault-id=dev@.passvaults extravaulty.yml
chocolate:banana

As I understand it, when you use --vault-id=dev@.passvaults you are telling it to use dev as a hint (not the vault password) and .passvaults as the file where the encrypt/decrypt password is.
This means that you can also do this:
root@2ca77340d571:/# ansible-vault view --vault-id=prod@.passvaults extravaulty.yml
chocolate:banana

Or this
root@2ca77340d571:/# ansible-vault view --vault-id=.passvaults extravaulty.yml
chocolate:banana

Which is not intuitive to me.
Does this help?


I want to scale up/down as fast as AWS Lambda but also be able to allocate the vCPU (minimum 8) per task, any advice?

You can check out AWS Fargate, I've just used it for a similar use case.

Setup:

Jobs are posted to an SQS queue where a scheduler lambda picks them up and launches an AWS Fargate task.

Caveats:

Lambda handles the SQS messages so you can't hand over the messages to Fargate (Lambda will auto-delete the messages from the queue once the lambda returns successfully), instead you'll need to handle retries via a separate lambda that gets notified whenever a task ends.
Fargate only supports up to 4 vCPUs
You need to pass any data to the container either via environment variables or as command line arguments

Launch times are not as fast as lambda, but much better than Batch and EC2 - I typically get my containers within 30 seconds of the lambda launching them.

Dead-letter/retry handling is a bit of a nuisance, but it's not that hard to implement as long as your process is well-behaved. You can easily get the process exit status from the API once the task has finished and you can store the number of retries as a tag on the task since you'll be launching a new task for each retry.

The scheduler and retry handler should be fairly trivial to write and if you need to pass larger amounts of data you can store it on S3 and send the location as an argument to your container.

A similar option - if you don't mind paying for the extra, unused capacity is to use ECS instead of Fargate and have unused capacity in the cluster so you can spawn new containers more or less instantly. This removes the limit on vCPU but you'll instead need to manage the cluster size (can be done via Autoscaling) and pay for the unused capacity.


I assume you talk about jenkins cli?
There is no command to abort the safeRestart.
Use quiteDown, cancelQuietDown instead and safeRestart only if needed.

safeRestart: Puts Jenkins into the
quiet mode, wait for existing builds to be completed, and then restart
Jenkins
quietDown: Put Jenkins in a Quiet mode, in preparation for a restart. In that mode Jenkins don’t start any build
cancelQuietDown: Cancel the effect of the “quiet-down” command


How to run a single one-time command on a kubernetes instance/pod

A little more information is needed about specific operations you are doing, but generally your options are:

For one-time operations: use kubectl exec to get into containers and do commands (works similar to docker exec).

Also for one-time operations, spawn separate pods with shell access to interact with existing pods, something along the lines:

kubectl run -it --rm --restart=Never --image busybox tempbusybox -- sh

For reproducible (not one-time) operations, you may use kubernetes jobs.

In some cases you may use init containers for setup tasks.


How to pass Gitlab env var with sensitive characters to bash script?

Surprisingly indeed GitLab works weirdly with the env vars.
It evaluates them internally, though it shouldn't in my view.

Value set as (in project Settings > CI/CD > Variables): 'aaa$bbb*%'
becomes 'aaa*%'
(with single quotes left!)

Couple of of proof links:

https://gitlab.com/gitlab-org/gitlab-ce/issues/45173#note_101659865
https://gitlab.com/gitlab-org/gitlab-ce/issues/27436

Workarounds:

1)
You can duplicate dollar ($) sign, i.e.: aaa$$bbb*%
(as mentioned on the first link above).

2)
Use base64 to encode and decode the value.

Do this first:

$ echo 'aaa$bbb*%' | base64
YWFhJGJiYiolCg==

Then store this value in project Variables.

And then in your script decode it:

PASS=$(echo ${sensitive_var} | base64 -D)


java.lang.UnsupportedOperationException: no known implementation of class jenkins.tasks.SimpleBuildWrapper is named BuildUser in jenkins

Have you installed the Build User Vars Plugin? This plugin provides the "BuildUser" class.


How can I persuade developers on my team to embrace "You build it, you run it"?

I think the easiest way is to change their performance goals so they are based off reliability as well as delivering code. Sell it as the company cannot succeed without both so why should the developers only be measured on one? The best way for them then to meet their performance goals is to be involved in operations.

Ultimately you need to convince them this is the best way for the company to succeed and therefore for them to succeed. That's hard and you can't expect them to be onboard from the start. They need to be sold on the value too.


multiple projects per EC2 with Load Balancers

You can archive that, but you will need to use the Application Load Balancer (ALB) which allows you to create rules (like HTTP URL paths, host header conditionals, query strings, HTTP method) to forward the traffic to a specific pool of servers that on your case would be those 2 EC2 instances.

e.g.:

/app1 --> :/app1
/app2 --> :/app2

Notice that you should have the ALB rule like this: /app1/* and /app2/*


As per https://12factor.net/config, configuration like this should be stored as environment variables.

  The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

A service that you connect to is suitable to be defined as configuration, so the use of env vars is appropriate here.


Running Ansible no-op's faster with a dynamic inventory and cached facts files

Instead of storing the Ansible code centrally, one could extract the code and distribute it to every project. Another benefit of this approach is that config and code will be seen together and that developers will change config and Ops could fix e.g. bugs instead of working in silos. Running the code per project will prevent that the executions will take longer, longer and longer.


What is a reasonable level of coding expertise expected of an SRE?

If an engineer is a great developer, why would he or she opt for an SRE role?
Here is a caricature of being a developer: Working in Sprints, getting a ticket with all of the BA work completed, spending 3-4 days on that problem, marking it done, getting the next ticket and so on. This is a really boring work environment.  Yes there are better environments to work in, and not all development roles fall into the scrum mediocrity trap, but it happens frequently.
DevOps is new enough that it's not yet been "solved" as a problem.  It certainly hasn't yet had all of the engineering challenge managed into separate roles, so it allows a certain type of developer a bit more freedom, working using multiple skills, in a fast paced environment. It's not for the faint hearted.
What is a reasonable expectation for an SRE when it comes to coding?
That they are a programmer that you would want on the development team. SRE is not a place for substandard developers. The development challenges in SRE are as great as that in the development space, but the space for errors is less.
Is it practical to have two cadres of SREs -- development focused ones or "SWE SREs" and operations focused ones or "Operation SREs"?
Not really. To be in SRE you must understand Dev and Ops. If thats not the case, then you are not doing SRE, you are doing something else.  It doesn't mean it's wrong, or not working for you - it could be just what you need for your environment, and you have the people that make it work. It is possible to have a specialist who isn't really a developer on the team, IF they bring an important skill to the team that the developers are not able to cover (e.g. low level networking or linux kernel knowledge), but this is done rarely.
Could they be a part of the same team?
You can place Dev and Ops into the same team.  It would make for some interesting management challenges. Please run the team in a more kanban style, ops doesn't work in sprints
Is it practical to rotate people between these two roles?
Most developers benefit from an exposure to the land of DevOps.  However, if it is a true ops environment, that timescale is probably 2-3 days at most.  Not sure most ops people would be able to assist meaningfully as a developer, but they would often make a good tester. Most ops people would not want to be in a testing role for long.


Pass information between Jenkins Job builds

There's a simple two step process...
In your originating job, tick step [ X ] Archive the artifacts and set the files to archive.

The files are copied to the master, stored in the jobs folder tree, under the builds directory for each build, along with the log. (Delete the build and it deletes the artifacts that came from it).
Install the Copy Artifact plugin.
Configure the job accordingly.

Despite the plugin's screenshot example, this is actually a bad practice for artifacts like jars. Those should be properly managed using an artifact repository like Nexus or JFrog.
See my response regarding keeping the the Jenkins space under control, managing the build and artifact retention soundly.


How to resolve the host operating system 'localhost' address from within a container?

You are correct in that using localhost within the container is referencing the container, and not your network.  You could use inspect, find the ip, and hardcode the ip.  However, a simpler and cleaner solution would be to utilize Docker's network features.  When you run the containers you can put the containers inside a defined network.  Containers in this network can resolve requests to other containers using their container name instead of an ip (e.g. Server=container2).

Docker Network Documentation


Configure Kubernetes with docker container on Ubuntu VM

You have to push your image in a registry in order to get the deployment working. Kubernetes does not deal with images (i.e. building, pushing...).

Try kubectl describe deployment mojo-deployment, you'll get more informations about why it's pending (surely because it does not know how to get your Docker image).


If instead of using the command line:
ansible-playbook -u root --private -key ~ / .ssh / id_rsa centos7.yml -i xxx.xxx.xxx.xxx,

I use:

ansible-playbook -u root centos7.yml -i xxx.xxx.xxx.xxx,

It works.


How to save groovy output to pipeline variable

Have you tried setting returnStdout to true in the sh() command per the Jenkins documentation?

You can then assign that output to a variable within your pipeline:

def artifactsList = sh(label: 'Running amd_distribution_input transformation', returnStdout: true, script: 'python amd_distribution_input_transformation.py')

and check what kind of object that variable is, and act on it accordinly

echo(artifactsList.getClass())


Is there a reason not to use Bamboo to run repetitive tasks instead of cron?

Using task schedulers like Bamboo or Airflow have a lot of advantages over cron, particularly in their ability to recognize errors and process dependent tasks accordingly.  By far the most common reason not to use them is that you simply haven't set up the infrastructure yet: some form of cron is installed on every Linux machine, and it's easy to add a new job in, whereas these more advanced schedulers require setting up fleets of machines, installing and configuring various pieces of software, and learning how to write new tasks.

Assuming that you already have this in place, though, there are a few usecases that cron handles better:

Running a job on a specific machine.  You can do this with other schedulers, but they're generally designed for tasks that can be run on any one of a set of machines; when you want to do something like run a log rotation on every server in your fleet, that's much easier to set up with cron.
Running time-sensitive tasks at exact times.  Distributed schedulers will schedule a job to run, but it may not actually run for some period of time, depending on what else is in the queue.  Cron doesn't have a queue, and so a task that needs to run every minute will run every minute (or at least as close to that as the OS scheduler will allow).


When I navigate to the URL, there do not appear to be any packages in the repo. Consider trying a mirror like: mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64


Why does the datasource "external" always fail?

The command being run by the external provider here is equivalent to the following:

bash "export ENC_HOOK_URL=$(aws --profile ${AWS_PROFILE} \
kms encrypt --key-id ${KMS_ALIAS} --plaintext ${HOOK_URL} --output json \
--region ${AWS_REGION})"

Bash is failing here because when used in this way it expects its first argument to be the filename of a script to run.

The -c option changes this interpretation so that it will instead expect this argument to be an inline script to run, which seems to be what you intended here. This could be expressed in the Terraform configuration like this:

data "external" "encrypt_url" {
  program = ["bash", "-c", "export ENC_HOOK_URL=$(aws --profile $${AWS_PROFILE} kms encrypt --key-id $${KMS_ALIAS} --plaintext $${HOOK_URL} --output json --region $${AWS_REGION})"]
}

Along with adding the -c option, it also seems like this script doesn't actually do anything, instead just assigning a result to a variable. Given that you asked the AWS command to produce JSON, I think perhaps the following will do what you want:

data "external" "encrypt_url" {
  program = ["bash", "-c", "aws --profile $${AWS_PROFILE} kms encrypt --key-id $${KMS_ALIAS} --plaintext $${HOOK_URL} --output json --region $${AWS_REGION}"]
}

In the interests of readability though, I'd recommend moving this command into an external script file (e.g. encrypt_url.sh) and then referencing it from the config:

#!/usr/bin/env bash

aws --profile ${AWS_PROFILE} \
    kms encrypt --key-id ${KMS_ALIAS} \
                --plaintext ${HOOK_URL} \
                --output json \
                --region ${AWS_REGION}

...

data "external" "encrypt_url" {
  program = ["bash", "${path.module}/encrypt_url.sh"]
}

This avoids the need for escaping the dollar signs, allows you to edit it in your text editor's shell script highlighting mode, and allows the script to be easily run outside of Terraform for testing purposes.


Multiple git profiles especialy with Atom

You'd better add this CA to a bundle of valid CA and keep using it for all cases with

git config --add --global http.sslcainfo=/path/to/cabundle
git config --add --global http.sslbackend=openssl

Using https://curl.haxx.se/docs/caextract.html as starting CA bundle sounds a good idea in my opinion.


Azure Pipelines: Exclude folders using Azure App Service Deploy

As @yang-shen-msft notes on Stack Overflow, there doesn't appear to be a way to honor the MSDeploySkipRules defined in the csproj file. Instead, files and folders can be skipped by defining the Additional Arguments (AdditionalArguments) parameter of the Azure App Service Deploy (AzureRmWebAppDeployment) task, as discussed in another Stack Overflow answer.
Since there doesn't appear to be any official documentation for the -skip rules, and the MSDeploy.exe documentation that Azure Pipelines references is out-of-date, the following provides additional details.
First, it's useful to recognize that when you deploy a project via Visual Studio, it's simply taking the MSDeploySkipRules configured in the csproj file and adding them to it's internal call of msdeploy.exe as -skip rules. So, given the following rule defined in the csproj:
<Project Sdk="Microsoft.NET.Sdk.Web">
  …
  <ItemGroup>
    <MsDeploySkipRules Include="CustomSkipFolder">
      <ObjectName>dirPath</ObjectName>
      <AbsolutePath>wwwroot\\Uploads</AbsolutePath>
    </MsDeploySkipRules>
  </ItemGroup>
</Project>

The -skip rule is interpreted as:
msdeploy.exe -skip:objectName=dirPath,absolutePath=wwwroot\\Uploads

Translating this to the Azure App Service Deploy (AzureRmWebAppDeployment) task, the resulting yml might look like:
- task: AzureRmWebAppDeployment@4
  inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: 'Azure Subscription'
    appType: 'webApp'
    WebAppName: 'MyStagingSite'
    packageForLinux: '$(Build.ArtifactStagingDirectory)/**/*.zip'
    enableCustomDeployment: true
    DeploymentType: 'webDeploy'
    enableXmlTransform: false
    enableXmlVariableSubstitution: true
    RemoveAdditionalFilesFlag: true
    AdditionalArguments: '-skip:objectName=dirPath,absolutePath=wwwroot\\Uploads'

Note: Multiple -skip rules can be defined on the same msdeploy.exe call.

Unfortunately, as mentioned above, there doesn't appear to be any official, first-party documentation for the -skip rules on msdeploy.exe. The 2014 documentation acknowledges them, and provides two examples, but doesn't expand on the options. That said, way back in 2012, @richard-szalay wrote a useful article, "Demystifying MSDeploy skip rules", which provides useful direction here for anyone requiring additional control over their -skip rules.


How do I get dict key from ansible playbook variable?

If I understand your question correctly you want to pass in a domain to find what could be a physical location or data center. I hope this is helps you!

roles/test/vars/main.yml

---
domains:
  gh4: [127.0.0.1,127.0.0.2,127.0.0.3,127.0.0.4,127.0.0.5]
  gh5: [127.0.0.7,127.0.0.8]
  gh6: [127.0.0.10,127.0.0.11,127.0.0.12]

roles/test/tasks/main.yml

---
# Make sure we have a domain passed in
- name: Ensure domain is defined
  assert: { that: domain is defined }

# Obtain the IP
- name: Simple A record (IPV4 address) lookup for {{domain}}
  set_fact:
      IP: "{{ lookup('dig', '+short', '{{domain}}')}}"
  register: output
  tags: test

# Check if the IP belongs to any location
- name: Checking location of IP
  set_fact:
    domain_location: "{{item}}"
  when:
     - IP in domains[item]
  with_items: "{{domains}}"

- debug:
    msg: "domain.com is in gh4.allserevers.com{{domain_location}}"
  when: domain_location is defined

Example Run
With domain.com pointing to 127.0.0.12

root@DESKTOP-U4VN2CS:/mnt/c/ansible/roles# ansible-playbook ../playbooks/test.yml  -e domain=domain.com
PLAY [127.0.0.1] ****************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************
ok: [localhost]

TASK [test : Ensure domain is defined] ******************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [test : Simple A record (IPV4 address) lookup for domain.com] **************************************************
ok: [localhost]

TASK [test : Checking location of IP] *******************************************************************************
skipping: [localhost] => (item=gh4)
skipping: [localhost] => (item=gh5)
ok: [localhost] => (item=gh6)

TASK [test : debug] *************************************************************************************************
ok: [localhost] => {
    "msg": "domain.com is in gh6.allserevers.com"
}

PLAY RECAP **********************************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0


How to test provisioning and configuration in Ansible setup?

Some options out there..

Testing tools: Sorted by github stars

Serverspec - Ruby, most popular tool out there, built on ruby's rspec
Goss - YAML, simple, <10MB self-contained binary, extremely fast, can generate tests from system state
Inspec - Ruby, think of it as an improved serverspec, almost same syntax, made by the chef guys. Built to be easier to extend than serverspec
Testinfra - Python, has the cool feature of being able to use Ansible's inventory/vars

Major differences between them:

Ultimately, I would suggest spending a day experimenting with all of them to get a feel for them before deciding for yourself.

With the exception of Goss, all the frameworks can run against a remote machine (ex. over ssh). Goss only runs locally or in docker w/ dgoss.
All frameworks can be run locally on a server, but require Python or Ruby to be installed or embedded. Inspec provides a self-contained <150MB bundle with an embedded version of ruby. Goss is a single <10MB binary with no external dependencies.
Goss has built in support for nagios/sensu output, this allows for easier integration with monitoring tools.
Goss tests tend to be simpler, but less flexible since it's based on YAML. Other frameworks allow you to leverage the full power of the underlying language Python/Ruby to write tests or extend the tool's functionality. (simplicity vs flexibility)
Goss allows you to generate tests from current system state
Testinfra to my knowledge is the only one that has built-in support for ansible inventory and variables
Inspec is backed by Chef

Continuous/divergence testing:

Chef Compliance - works with inspec to continuously test your servers, paid product
Goss - Can be easily hooked into Nagios or Sensu. Also, supports exposing server tests as an http endpoint.

Testing harnesses for development:

kitchen - Testing harness tool, launches instance, runs config management code, runs test suite. Made by the chef guys
Molecule - Similar to test kitchen, but written specifically for ansible

Full Disclosure: I'm the author of goss

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.


Jenkins API call (or Groovy builtin) that returns list of all currently running builds of jobs

Turns out you can hit the following URL to get an XML output of all currently running builds, including their build numbers.

http://jenkinsURL/computer/api/xml?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds

I greped it for a count and used that for my calculations.


What do we know about how open source software becomes eligible for the Docker Hub library?

From https://docs.docker.com/docker-hub/official_repos/

  Docker, Inc. sponsors a dedicated team that is responsible for reviewing and publishing all content in the official repositories. This team works in collaboration with upstream software maintainers, security experts, and the broader Docker community. While it is preferable to have upstream software authors maintaining their corresponding Official Repositories, this is not a strict requirement.


What are the main differences between Jenkins and TeamCity if one is used to working with Jenkins?

TeamCity :

It does look nicer, if this is important for your team then it should definitively weight in.  That said, if it is VERY important than you probably will end up creating tools or some sort of dashboard overlay to support your team at which point what you really want is the one with the best API.  Have not tried Jenkins API so I cannot compare, the TC API though should get you what you need.

Their support is pretty good, the guys respond relatively fast and are courteous.  However that does not mean you will get what you want.  If you use the system in an unconventional way you can very well get your bug put on the shelves... happened to us.  At which point it becomes quite frustrating to use, you are faced with a black box leaving little alternative than to work around it.  At this point things can get hackish and ugly.

It is generally pretty quick to get done what you want and the log interaction quasi API is a very nice feature if you do a lot of custom scripting in your pipeline.

Jenkins:

Battle tested and Extensive.

But it is somewhat less pretty, would not go as far as saying it is ugly though, one could say that functionality comes before appearance.

I'm pretty sure you can find private paid support plan by third party companies if you look around.  If this is important for your shop don't just block at the "OpenSource" part of the deal, the community is quite expansive.

Lots, I mean LOTS of plugins.  Again don't just just limit yourself to the official channels, lot more plugins can be found in github and other places.

I found both to be about equally fast to get started with, though with Jenkins you might need to be more dynamic with plugins than with TeamCity.  So if you have a tight IT department and wont get admin access to the server this could pose a problem.  Compounded by their much faster release cycle than TeamCity (weekly).

I found that Jenkins does support more release cycle paradigms than TeamCity.  Might be easier to find a process template out of the box that is a closer match to what you have in mind. I say this with reserves as I have not dealt with TeamCity for 2 years now.

I personally prefer Jenkins mostly because I am biased towards Open Source for such tools and in a lesser degree because I found it's structure and config mechanism agree more with me.

YMMV


Turns out -}} is a special syntax which says 'eat up any following whitespace'.  Removing the - fixed this.

I get a blank line, but don't really care, considering!


AWS CloudWatch Logs Publisher plugin hangs on job completion

Googling returns only this Q&A:

It looks like that it is not a reported issue. Also combined with your remark, i.e.:

  It always completes successfully, even with the delay.

I wonder whether it is really an issue as I have seen a lot of logs and also weird messages while everything worked. What I do in such situation is talk with the developer of the code. In this case I would create an issue in the Jenkins bug tracker and discuss whether this is an issue and whether the developers should change the code that triggers this message.


Who redirect ports on Windows to access http container?

it uses the Hyper-V Virtual Switch (https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/container-networking)


How to explicitly kill the process with PID 1 from inside a container?

PID 1 is a special PID which will ignore certain signals unless handlers are explicitly created. Docker has some documentation on this:

  Note: A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. So, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.

From here: https://docs.docker.com/engine/reference/run/#foreground

To kill PID 1 you will have to explicitly declare the handler for the SIGTERM signal or, in current versions of Docker, pass the --init flag in the docker run command to instrument tini.


Application calling AWS internal load balancer in same subnet is timing out

So what you have actually is this:

As your API ELB is in a private zone it can't be accessed from the internet.
Your frontend in React.js just run in User's browser and not on the UI servers, those server just serve static files.

You have two options, configure your frontend servers to redirect API calls to the API ELB or just update the API ELB to be internet facing.

The usual pitfall of JavaScript apps is forgetting they run inside user's browser and not on the frontend servers as a JEE application would.


Securely saving AZURE_STORAGE_KEY and storage account name in Linux systems

Instead of storing them as variables in the bash script themselves, can you store them as environment variables on the system from which the bash script is executing? This is a pretty common practice that will allow your scripts to execute on different systems without having to modify the script on each and every one.

Not sure what you mean by the AWS cli "saves the key and the secret in encrypted format", but if you look at the credentials file in ~/.aws, your access key and secret access key are both in plain text. I'm not sure how the Azure cli manages profiles, but AWS uses the AWS_PROFILE environment variable to distinguish which account the cli should execute as.


How to restrict filesystem access in Atlassian Bamboo builds?

There is currently no ability to restrict jobs which are able to run on the same agent from potentially interacting with each other. There are a bunch of feature requests asking for this sort of granularity, but if I understand your question correctly the most fitting request would be this one BAM-2504 Jira Ticket

It's a huge gap in the product line, the only solution I've found is similar to what is proposed by the request linked above, essentially your bamboo process would need to run with sufficient privileges to impersonate a set of users who represent the projects which you would like to keep separate.

Once you have this sort of mechanism in place you just need to try to enforce that all plans run as one of the impersonatable accounts, depending for example on the project or the creator etc.

Problematically the way that access controls currently exist this would mean that few core admins would need to set up all the plans so that they can he sure of the required permissions split rather than letting non-administrative users edit and create their own plans.

If this sort of approach isnt feasible, which it isn't once you get into the "many hundreds of users" type of range, then the only thing you can really do to try and stop build jobs from interacting with each other is implement the very weak controls that bamboo gives you.

I've tried two approaches to doing this:

Delete or cripple (remove all capabilities from) your local agents and for every different project/team/whatever that on boards to your bamboo instance you need to force them to BYO build server. In most instances I've been involved in the cost of an agent is utterly trivial compared to the cost of potential data leakage or malicious plan interactions.
Make sure that projects that do have, or that think they do have, sensitive data involved in their plans always sanitise their environments after they build. This shifts the burden off the team administering the tools and onto the projects writing their plans and forces them to defensively clean up any information that they don't want potentially available to others.

Neither solution is even close to perfect but to the best of my knowledge that's about as much separation as you can enforce if you have a shared bamboo instance.


See which Roles contain a GCLOUD Permisson

Looks like only the roles/billing.admin:
https://cloud.google.com/iam/docs/understanding-roles#billing-roles


How can I use bash functions in makefile?

One possible issue is that the NVM install script only writes the commands to set itself up to either .bash_profile OR .bashrc. Depending on your systems configuration, it may only be setup for interactive shells, not ones spawned by other programs (more info). Sourcing the nvm setup script can be a good option, as a previous comment noted. If you have a number of commands to run for 1 make target, look into .ONESHELL. Sourcing $NVM_DIR/nvm.sh should take care of it.
Another option that invokes bash directly with the interactive flag might work as well.


Kubernetes - Deploying older version of app in new API

Prior to apps/v1 the selector field was implicitly picked up from the pod template's labels. So in your case it would be:

spec:
  selector:
    matchLabels:
      io.kompose.service: web

Starting with apps/v1 an explicit selector field is required by the Kubernetes API.

I would suggest not to change the iptables on your own as it it the responsibility of your Kubernetes network plugin and is most likely done well.

As far as I am concerned you may simple update the Deployment manifest and no uninstall is required.


There is currently no full solution to this integrated into Terraform, but there are some building blocks that could be useful to assist in writing tests in a separate programming language.

Terraform produces state files in JSON format that can, in principle, be used by external programs to extract certain data about what Terraform created. While this format is not yet considered officially stable, in practice it changes infrequently enough that people have successfully integrated with it, accepting that they might need to make adjustments as they upgrade Terraform.

What strategy is appropriate here will depend a lot on what exactly you want to test. For example:

In an environment that's spinning up virtual servers, tools like Serverspec can be used to run tests from the perspective of these servers. This can either be run separately from Terraform using some out-of-band process, or as part of the Terraform apply using the remote-exec provisioner. This allows verification of questions like "can the server reach the database?", but is not suitable for questions such as "is the instance's security group restrictive enough?", since robustly checking that requires accessing data from outside of the instance itself.
It's possible to write tests using an existing test framework (such as RSpec for Ruby, unittest for Python, etc) which gather relevant resource ids or addresses from the Terraform state file and then use the relevant platform's SDK to retrieve data about the resources and assert that they are set up as expected. This is a more general form of the previous idea, running the tests from the perspective of a host outside of the infrastructure under test, and can thus collect a broader set of data to make assertions on.
For more modest needs, one can choose to trust that the Terraform state is an accurate representation of reality (a valid assumption in many cases) and simply assert directly on that. This is most appropriate for simple "lint-like" cases, such as verifying that the correct resource tagging scheme is being followed for cost-allocation purposes.

There is some more discussion about this in a relevant Terraform Github issue.

In the latest versions of Terraform it is strongly recommended to use a remote backend for any non-toy application, but that means that the state data is not directly available on local disk. However, a snapshot of it can be retrieved from the remote backend using the terraform state pull command, which prints the JSON-formatted state data to stdout so it can be captured and parsed by a calling program.


Six repositories as a single deployment Docker+CircleCI

With docker compose (or docker swarm if you want to distribute the containers) create a container for each Git repository. This should map well to docker’s philosophy of each container being used for a single purpose.

As far as examples go, I can recommend you have a look at veleda (shameless self-promotion) which combines 3 databases, a flask app, a ruby app (Grafana), Jekyll and nginx. It also includes unit tests that are integrated into Travis-ci and should be transferable to circle-ci.

Regarding the nginx aspect, 030 already mentioned that there are finished images to make your life easier. Even ones that include automated Let’s Encrypt services.

Ansible can be integrated well into the mix. If you used it to install dependencies and network configurations, this is now handled by docker. For SSH-based tasks, ansible remains useful. Check out this comparison of tools and their scope’s of responsibility.


Docker fails to mount volume inside Jenkins scripted pipeline

In configuration settings of Docker plugin for Jenkins, there were missing mountings for the volumes, because I was deploying docker images within docker slave container, so that the "master" docker container (with Jenkins slave agent) couldn't access the real files from the host machine.


get value of default_statistics_target on postrgesql

Actually, there are two questions:

Where default(100) value is declared?
How can I see Stats target value before applying SET default_statistics_target=1000

Where default(100) value is declared?

You can see it in the source code

{
      {"default_statistics_target", PGC_USERSET, QUERY_TUNING_OTHER,
          gettext_noop("Sets the default statistics target."),
          gettext_noop("This applies to table columns that have not had a "
                       "column-specific target set via ALTER TABLE SET STATISTICS.")
      },
      &default_statistics_target,
      100, 1, 10000,
      NULL, NULL, NULL
  },

How can I see Stats target value before applying SET default_statistics_target=1000 ?

Let's suppose that column bar of table foo has  attstattarget=500:

alter table foo alter column bar set statistics 500;

So, there are two ways to show attstattarget

SQL_query:

SELECT attrelid::regclass, attname, attstattarget FROM pg_attribute WHERE attstattarget > 0 order by attstattarget desc;

 attrelid | attname | attstattarget
----------+---------+---------------
 foo      | bar     |           500

\d+ command:

\d+ foo

                                                            Table "public.foo"
          Column           |            Type             |                  Modifiers                   | Storage  | Stats target | Description
---------------------------+-----------------------------+----------------------------------------------+----------+--------------+-------------
 bar                       | varchar                     | not null                                     | plain    | 500          |

** Note**: Postgres doesn't show stats target until it distinct from default value.

300 - yet another magic number

Thx to this comment, you can read a paper about another magic number and check it in the source code:

/*
   * Determine which standard statistics algorithm to use
   */
  if (OidIsValid(eqopr) && OidIsValid(ltopr))
  {
      /* Seems to be a scalar datatype */
      stats->compute_stats = compute_scalar_stats;
      /*--------------------
       * The following choice of minrows is based on the paper
       * "Random sampling for histogram construction: how much is enough?"
       * by Surajit Chaudhuri, Rajeev Motwani and Vivek Narasayya, in
       * Proceedings of ACM SIGMOD International Conference on Management
       * of Data, 1998, Pages 436-447.  Their Corollary 1 to Theorem 5
       * says that for table size n, histogram size k, maximum relative
       * error in bin size f, and error probability gamma, the minimum
       * random sample size is
       *      r = 4 * k * ln(2*n/gamma) / f^2
       * Taking f = 0.5, gamma = 0.01, n = 10^6 rows, we obtain
       *      r = 305.82 * k
       * Note that because of the log function, the dependence on n is
       * quite weak; even at n = 10^12, a 300*k sample gives <= 0.66
       * bin size error with probability 0.99.  So there's no real need to
       * scale for n, which is a good thing because we don't necessarily
       * know it at this point.
       *--------------------
       */
      stats->minrows = 300 * attr->attstattarget;
  }
  else if (OidIsValid(eqopr))
  {
      /* We can still recognize distinct values */
      stats->compute_stats = compute_distinct_stats;
      /* Might as well use the same minrows as above */
      stats->minrows = 300 * attr->attstattarget;
  }
  else
  {
      /* Can't do much but the trivial stuff */
      stats->compute_stats = compute_trivial_stats;
      /* Might as well use the same minrows as above */
      stats->minrows = 300 * attr->attstattarget;
  }


Getting permission denied with docker pipeline on jenkins

You need to add the jenkins user to the docker group:

# run the following command as root
usermod -aG docker jenkins


How to scale-down in a multi-tenant environment?

Autoscaling is a good case for machine learning

This is a hard problem to do well.

What you really want is something like Nest Thermostat for your EC2 infrastructure.

There are (aforementioned) multiple dimensions of resource demand/limitations.

CPU
memory
disk space
disk IO
network IO
concurrency/latency/queue depth

There are multiple indicators of demand (in addition to the above).

concurrent unique visits/sessions
pages per visit/session
rate of engagement/interaction feature usage

There are multiple common patterns of changing demand over time.

daily user demand cycles
weekly user demand cycles
monthly ...
annual ...
special event/days
DDoS load
media/marketing exposure traffic spikes

There are multiple financial decision factors.

does revenue scale with traffic? How? (How conservative do you need to be?)
is there a hidden cost to control (transaction costs, limits)?
what's the cost model of scaling? (things in a pipeline scale together, things in a load-balanced cluster scale independently)

Before long, if you try to hand-optimize on hand-selected features you're going to have a monster of technical debt that is possibly more complicated than any other logic in your site. Amazon makes more money when you err (with a large margin) on the side of caution, so their tools will probably never get close to what you want.

Instead, choose an architecture/technology stack that can grow/scale so you don't have to get it exactly right the first time. Then pick a few factors which you think are obvious. Then try to come up with a way to sort multiple representative possibilities in order of preference. Then collect some real world data covering all those points. If you're lucky, a simple obvious hand-coded solution will jump out at you from looking at the data. If not, code up something that will give you an approximate model f(x1,x2,x3,x4) --> y * app nodes, using an appropriate algorithm.

I bet you didn't think this one was going to be so much fun!


How to pin OS package versions in Docker image

I'm not sure that this qualifies as an "easier" way, but you could host the specific packages on a server (e.g., Artifactory, S3).  You would then have the Dockerfile connect to this location and use the packages from there.

This would allow you to avoid creating a custom base image and you would not have to manually download the packages.


Scaling replication controllers via API in Kubernetes

After long playing with it and consulting on slack DevOps chat, I was able to figure it out. Here is a workable curl:

curl -sSk -H \"Authorization: Bearer ${KUBE_TOKEN}\" -H 'Content-Type: application/merge-patch+json' -X 'PATCH' $URL_NAME/api/v1/namespaces/kube-system/replicationcontrollers/kube-registry-v0 -d '{"spec": {"replicas":  1}}'

One of the important components here is "Content-Type: application/merge-patch+json" Without it specification, Kubernetes will not recognize data in API request correctly.


Running Two infinite commands on single pod/image

Though it's not advised to have multiple processes inside a container, but you have 2 options:

Use a process manager like supervisord or wrapper script to use as CMD (meta code)

start_first_process_in_bg
if START_FAILED
  exit
start_second_process_in_bg
if START_FAILED
  exit

while sleep 5; do
  check_first_process
  check_second_process

  if ANY_PROCESS_FAILED; then
    exit


Continuous delivery workflow: Build script for PHP Composer (Drupal) on Docker

Each CI/CD system is different when it comes to performing the job(s) that the execution pipeline consists of. In most (if not all) cases the details are part of a configuration which is specific to the particular CI/CD system used. As you mentioned Jenkins uses a Jenkinsfile, GitLabCI uses .gitlab-ci.yml, others use "live"/GUI-based configurations, etc.

So I don't think you can find a common solution directly pluggable in an arbitrary CI/CD system. But you can get close to that: fundamentally each of these configurations are simply adaptation layers between the CI/CD system and the automated task(s) to be performed.

So if you can create a (set of) standalone script(s) to perform your automated step(s) in as portable way you need as possible you should be able to integrate with in almost any CI/CD system without too much trouble: any decent CI/CD system should support a custom script invocation mode to cover for whatever special case that doesn't properly fit into its pre-defined cases.

Using python/bash is generally quite portable, up to you to decide if that indeed covers all possible execution environments you'd be willing to consider.

I'd recommend structuring such script itself as a wrapper around smaller scripts implementing its logical/functional steps - some CI systems may require individual control of the steps or may offer advantages if such control is provided. In your case I'd have separate scripts for:

checking out the source code
importing the database from production
building the Drupal project running in a Docker container (maybe this can/should be further split if it makes sense?)

And I'd add a wrapper script tying them together for one-shot end-to-end execution, either by CI/CD systems not interested in the individual steps or even for manual invocation.


Best Practices for a multi environment deployment using Jenkins

This is a very good question as it is a common anti-pattern to couple ci/cd tools with environments.

Jenkins is a build factory. As such, it should be totally agnostic of the notion of environment, or even delivery.

The best pratice is to have some sort of staging process and / or interface (if you can afford it: a dedicated delivery software).

Staging process

You should try to create some sort of staging jobs for each environment, in which you'll define the configuration of the environment and bundle the final package with it.

Next you'll have delivery jobs for each environment.

Delivery tools

In order to do create the staging and delivery jobs, you may use very basic scripting tools, like shell, but there are more adapted scripting tools, like Ansible, puppet, chef...

If you can afford the expense, you may consider investing in a deployment software (I'll mention some that I worked with in the comments section).

Notice that those softwares manage some kind of environment staging process.

Obviously, it makes a lot of sens to combine deploy softwares and scripting.

Beyond segregating tools and environment

Since you asked for best practices, it appears to me that it is worth mentioning another commonly found anti-pattern: coupling SCM tools and delivery process.

It is a very good practice to store environment configuration (not passwords or confidential informations, of course) and go live scripts in SCM tools (like svn or git...). It is however a very bad practice to check-out environment configuration and go live scripts DURING the go live. It might simply be unavailable when you need it.

This check-out phase should be part of the staging process that I mentioned before.

Idem potence

Another best practice is that your scripts should be idem-potent: this means that you should be able to play your scripts once to perform the staging and delivery. Then you should be able to play them again and again, and they would not change the state of your system unless something chacked in the configuration.

Scaling

As a final best practice I shall share from direct experice is scalling Jenkins: different teams have different needs and uses of Jenkins. When too many teams share a common Jenkins, there might be issues with resources. The worst case is when a team requires to restart the Jenkins master while another needs to deliver.

The ideal would be to have one Jenkins per team or per group of teams that share the same goal and delivery planning.


Which system user should I use to run Postgres?

What is the default user of postgresql?

postgres

  How does hardcoding the postgres user improve security?

It's not using the root user.

You can use any account you like, but consistency is a Wonderful Thing, especially when you have lots [and lots] of Postgres clusters to manage.

  What special rights does the postgres user have?

Not many.

It installs itself a hole through the local firewall but, otherwise, it's just another user.

  What is the upside and downside of using an unprivileged user ?

You should never run applications (or databases) as root (unless they actually need to be).

Always keep the biggest and best tools in the toolbox for yourself, ready to sort out the mess that Applications (or Users) make.  If you run stuff as root and it decides to do something really stupid to your machine, you may not be able to fix it.


terraform chef provisioning - cannot validate certificate for 10.90.239.223 because it doesn't contain any IP SANs

Answered on Slack, the likely first issue here is that the output from openssl req is not a cert, it's a CSR. But there are some other lingering issues here too. It is recommended to use something like Certstrap to do internal cert generation rather than raw openssl commands as the modern standards are quite fiddly.


The plugin you found doesn't have a 1:1 ratio between Heroku instance with build jobs since it takes the job name as parameter, so you should be able to run any number of jobs from a single Heroku instance.

As for passing a space in a job parameter, the word split is made on spaces so quoting won't change anything. You could possibly try to use the HTML spacing character %20 and see if it works?

Personally I would try to use the plugin to run the correct job with the associated parameters directly instead of going through filtering jobs.